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Abstract 


The  Department  of  Defense’s  utility  systems  have  suffered  from  under-funding 
and  lack  of  technology  investment,  thereby  creating  a  service  gap  between  these  DoD 
systems  and  systems  operated  by  commercial  providers.  In  order  to  overcome  this  gap 
and  refurbish  their  systems,  the  DoD  has  focused  its  efforts  on  privatization  of  these 
systems.  Although  the  Air  Force  has  published  source  selection  guidance  to  aid  the 
privatization  process,  the  utilities  privatization  decision  is  extremely  vital  and 
“permanent”  and  thus  warrants  backup  source  selection  methodologies. 

Since  awarding  contracts  in  the  utilities  privatization  arena  requires  a  decision 
considering  tradeoffs  between  multiple  competing  objectives  such  as  economic 
feasibility,  security,  readiness,  reliability,  and  cost,  a  value-focused  thinking  approach 
was  used  to  create  a  structured,  standardized  value  model  taking  into  account  the  values 
of  the  United  States  Air  Force  with  regard  to  utilities  privatization.  This  model  was  then 
used  to  evaluate  a  set  of  randomly  generated  contractor  proposals  for  leasing  an  Air  Force 
owned  electrical  system  and  perform  detenninistic  and  sensitivity  analysis  on  the 
recommended  decision  generated  by  the  model. 

The  results  of  this  research  provide  a  quantitative,  objective,  reliable,  and 
defendable  tool  for  a  utilities  privatization  source  selection  decision.  The  value  model  is 
generic  enough  for  evaluating  the  source  selection  decision  of  any  utility  system,  but  is 
able  to  accommodate  changes  in  guidance,  decision  makers,  and  other  factors. 
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THE  APPLICATION  OF  VALUE-FOCUSED  THINKING  TO  UTILITIES 


PRIVATIZATION  SOURCE  SELECTION 


Chapter  1.  Introduction 


1.1  Introduction 

Modern  civilizations  and  their  economies  are  built  upon  sufficiently  developed 
infrastructures  for  the  proper  supply  of  water,  gas,  electricity,  and  other  utilities. 
Therefore,  the  respective  utility  systems  that  supply  and  treat  water  and  provide 
electricity  and  natural  gas  are  intrinsically  entwined  with  human  societies  and  progress 
(de  Luca,  1997).  A  society  unable  to  provide  these  essential  utilities  risks  losing  its 
ability  to  function,  thus  it  is  imperative  that  effective  and  efficient  utility  systems  exist. 

Historically,  utility  systems  have  been  held  to  be  natural  monopolies  run  by 
privately  owned,  regulated  mega-corporations  or  government-run  companies  (Plaff, 

2001).  During  the  last  20  years,  however,  increasing  consumption,  rising  costs,  and 
ecological  considerations  have  made  it  difficult  for  companies  to  meet  new  demands  and 
implement  technological  advances;  this  has  fueled  motivation  to  create  more  efficient  and 
cost-effective  market  solutions  for  utility  systems  (de  Luca,  1997).  Many  of  the  privately 
owned  mega-corporations  have  been  deregulated  to  eliminate  bureaucratic  inefficiencies, 
thereby  creating  market  competition  in  efforts  to  stimulate  new  technologies  and 
investment.  However,  finding  ways  to  “force”  publicly  held  utility  systems  to  meet 
increased  demands  while  simultaneously  maintaining  efficiency,  equity,  and  long-term 
environmental  sustainability  has  been  more  troublesome  because  of  their  political 
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surroundings  (de  Luca,  1997;  Plaff,  2001).  Currently,  one  of  the  most  popular,  and  some 
would  argue  most  effective,  market  solutions  for  “fixing”  public  utility  systems  is 
privatization  (Plaff,  2001).  Privatization  is  simply  the  transfer  of  government-held 
systems  to  private  (commercial)  sector  companies;  proponents  of  believe  private  sector 
companies  can  more  efficiently  and  effectively  provide  the  services  offered  by  public 
utilities  (Prager  and  Desai,  1996).  However,  opponents  of  privatization  assert  that  public 
administrations  should  avoid  “knee-jerk  reactions”  in  claiming  that  privatization  is 
always  more  effective,  more  efficient,  and  less  costly  than  the  goods  and  services  being 
provided  by  government  entities  (Prager  and  Desai,  1996). 

1.2  Problem  Background 

The  Department  of  Defense  (DoD)  has  encountered  many  of  the  same 
bureaucracy-driven  problems  with  their  utility  systems.  The  DoD’s  utility  systems  have 
suffered  from  under-funding  and  lack  of  technology  investment,  which  has  created  a 
perceived  gap  between  the  service  provided  by  DoD  systems  and  systems  operated  by 
commercial  providers.  The  strong  belief  that  the  private  sector  is  “inherently  dynamic, 
productive,  and  dependable,”  along  with  the  high  capital  costs  associated  with  bringing 
existing  utility  systems  up  to  current  technological  standards,  has  convinced  many 
decision  makers  that  the  private  sector  can  provide  less  costly  service  at  a  higher  service 
level  than  current  publicly  owned  utilities  (Bamekov  and  Raffel,  1990;  Orwin,  1999;  de 
Luca,  1997).  Therefore,  the  DoD  has  focused  on  the  privatization  of  its  systems  to 
overcome  the  perceived  service  gap  (Marrs,  2003). 
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In  December  of  1997,  the  DoD  issued  Defense  Reform  Initiative  Directive  #9 
(DRID  #9)  to  direct  the  military  services  to  develop  a  plan  for  privatizing  all  utility 
systems  by  January  1,  2000,  except  where  privatization  of  a  system  was  either  not 
economically  or  not  appropriate  due  to  unique  security  reasons.  Each  service  was 
required  to  outline  their  requirements  for  accomplishing  utility  privatization,  including  a 
timetable  with  internal  benchmarks  (Deputy  Secretary  of  Defense,  1997).  However,  it 
was  soon  realized  that  the  number  of  systems  to  be  privatized  and  the  complexity  of  the 
privatization  process  made  this  goal  unrealistic.  Accordingly,  the  DoD  issued  DRID  #49 
to  extend  the  completion  goal  to  September  30,  2003.  It  also  directed  the  services  to 
observe  two  milestone  dates:  a  “Go/No-Go”  decision  for  all  utility  systems  by 
September  30,  2000,  regarding  whether  or  not  a  system  was  available  for  privatization 
(i.e.,  no  unique  security  requirements  and  economically  feasible),  and  a  release  of 
solicitations  for  all  systems  identified  as  available  by  September  30,  2001.  It  also  set 
forth  previously  unprovided  written  guidance  on  exempting  systems  from  privatization, 
competing  contracts,  and  conducting  economic  analyses  (Deputy  Secretary  of  Defense, 
1998). 

It  was  soon  realized  that  DRID  #49  was  overly  ambitious  as  well.  By  September 
of  2001,  more  than  1,300  utility  systems  were  under  solicitation  or  pending  the  release  of 
a  Request  for  Proposal  (RFP).  The  utility  industry  reported  that  this  was  saturating  the 
market  and  decreasing  effective  competition  for  the  award  of  these  privatization  efforts. 
The  military  departments  concurred  and  the  DoD  responded  by  releasing  revised 
guidance  on  utilities  privatization  in  October  of  2002.  The  revised  guidance  directed  the 
services  to  have  a  utilities  privatization  (UP)  evaluation  done  on  every  system  by 
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September  30,  2005,  to  determine  the  availability  of  each  system  for  privatization 
(Deputy  Secretary  of  Defense,  2002).  It  also  shifted  the  emphasis  away  from  a 
completion  date;  instead,  it  focused  on  the  process  itself.  Accordingly,  the  revised 
guidance  included  three  interim  milestones:  the  establishment  of  final  internal  milestones 
for  each  service  by  October  15,  2002;  the  closure  of  RFPs  or  the  submittal  of  certificates 
of  exemption  for  at  least  80  percent  of  the  systems  specified  as  available  for  privatization 
by  September  30,  2003;  and  finalization  of  Source  Selection  Authority  (SSA)  decisions 
or  the  submittal  of  certificates  of  exemption  for  at  least  65  percent  of  the  systems 
specified  as  available  for  privatization  by  September  30,  2004  (Deputy  Secretary  of 
Defense,  2002). 

To  help  its  personnel  meet  the  revised  UP  guidance,  the  Air  Force  released  its 
own  service-specific  guidance  in  October  of  2002  in  the  form  of  the  Air  Force  UP  Policy 
and  Guidance  Manual.  This  guidance  reviewed  applicable  policies,  defined  roles  and 
responsibilities,  and  specified  the  utility  privatization  process  through  contract  award  and 
property  transfer.  The  Air  Force  also  established  the  Air  Force  Utility  Privatization 
Executive  Steering  Group  in  May  of  2003  to  oversee  execution  of  the  Air  Force’s  utility 
privatization  program  (Department  of  the  Air  Force,  2002). 

To  meet  the  interim  goal  required  in  DRID  #49  of  releasing  Requests  for  Proposal 
(RFPs)  for  all  available  utility  systems  in  the  inventory  by  September  30,  2001,  the  Air 
Force  released  an  RFP  template  on  February  11,  2000.  After  its  initial  release,  the 
template  was  amended  in  June  of  2000  and  again  in  February  of  2003.  Released  by  the 
Air  Force  Civil  Engineer  Support  Agency  (AFCESA),  the  most  recent  revision  of  the 
template  provides  a  comprehensive  process  for  preparing  RFPs.  It  describes  how  a 
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contractor’s  price  schedule  should  be  based  on  inputs  such  as  the  purchase  price  of  the 
system,  the  cost  of  initial  capital  upgrades,  transition  costs,  etc.  It  also  covers  particular 
descriptions,  specifications,  and  work  statements  that  should  be  included  in  the  RFP  to 
encompass  a  wide  range  of  subjects  such  as:  service  requirements  and  performance 
standards,  employees,  security,  rights  of  way,  service  interruption  and  contingencies, 
work  order  response  requirements,  work  coordination  requirements,  environmental 
compliance,  upgrades,  renewals  and  replacements,  transition  plans,  and  specialty  training 
requirements.  The  template  also  specifies  which  clauses  from  various  regulations  are 
considered  mandatory  items  and  identifies  specific  contract  terms  and  contract 
administration  requirements.  The  RFP  template  also  lists  the  various  representations, 
certifications,  and  statements  required  of  the  contractors.  Finally,  the  template  includes  a 
detailed  section  on  how  a  contractor  should  prepare  its  proposal  (DESC,  2002). 

One  of  the  improvements  included  in  the  2003  revision  is  the  inclusion  of 
instructions  to  the  contractors  on  how  proposals  will  be  evaluated.  The  current 
evaluation  process  consists  of  five  factors:  Mission  Capability,  Past  Performance, 
Proposal  Risk,  Socioeconomic  Plan,  and  Price.  Each  of  the  factors,  other  than  Price,  is 
evaluated  against  a  scale  constructed  specifically  for  that  particular  factor  and  receives  an 
appropriate  rating.  The  evaluation  falls  under  the  auspices  of  the  Source  Selection 
Evaluation  Team  (SSET),  which  is  responsible  for  advising  the  Source  Selection 
Authority  (SSA)  and  Source  Selection  Authority  Board  (SSA  Board)  as  appropriate.  The 
SSET  is  also  responsible  for  performing  the  comparative  analysis  of  proposals  received 
as  well  as  conducting  the  evaluation  in  an  impartial  and  equitable  manner  (DESC,  2002). 
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1.3  Problem  Statement 


The  Air  Force  has  published  enormous  amounts  of  guidance  regarding  the 
privatization  process,  but  not  enough  focus  has  been  placed  on  developing  proposal 
evaluation  tools  for  use  in  the  source  selection  process  for  competitive  UP  situations. 
Although  an  economic  feasibility  program  had  been  created  and  guidance  was  available 
in  the  Air  Force  UP  Policy  and  Guidance  Manual,  no  standardized  methodology  for 
accomplishing  the  award,  implementation,  measurement,  and  verification  steps  exists. 
The  release  of  the  2003  revisions  to  the  RFP  template  introduced  a  methodology  for 
source  selection  evaluation.  This  methodology  is  a  step  in  the  right  direction;  however,  it 
does  not  eliminate  a  large  amount  of  the  subjectivity  present  in  the  decision  making 
process  and  it  has  no  quantitative  basis.  Additionally,  it  does  not  incorporate  some  of  the 
findings  from  recent  UP  research.  One  of  the  most  common  lessons  learned  in 
privatization  efforts  around  the  world  is  that  a  reliable,  objective,  repeatable,  and 
defendable  source  selection  process  is  absolutely  necessary  (Orwin,  1999;  de  Luca, 

1997).  Orwin  (1999)  states  that  not  only  is  a  standardized  source  selection  process 
needed  but,  because  of  the  enormous  potential  impact  of  these  contracts,  which  are 
awarded  for  timeframes  of  10  to  50  years,  a  backup  (or  secondary)  source  selection  tool 
should  also  be  used  to  ensure  the  “accuracy”  of  the  primary  tool  (Orwin,  1999). 

Since  the  Air  Force  has  been  mandated  to  accomplish  utilities  privatization,  a 
definitive  and  more  comprehensive  process  is  needed  to  ensure  that  base  utilities 
privatization  teams  have  a  reliable,  objective,  repeatable,  and  defendable  evaluation 
technique  for  awarding  contracts.  Therefore,  the  purpose  of  this  thesis  is  to  pursue  the 
development  of  a  standardized  model  that  can  be  used  as  a  secondary  source  selection 
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decision  support  tool  in  any  competitive  utilities  privatization  scenario  in  the  Air  Force. 
Because  of  time  and  scope  limitations,  however,  this  portion  of  the  research  will  be 
limited  to  developing  a  model  for  electrical  system  UP  source  selection  decisions. 

1.4  Research  Objective  and  Questions 

Awarding  contracts  in  the  utilities  privatization  arena  requires  a  decision  that 
considers  tradeoffs  between  multiple  competing  objectives  such  as  economic  feasibility, 
security  requirements,  readiness  requirements,  reliability,  service,  and  cost.  A  key  to  this 
decision  making  process  is  whether  a  structured  technique  exists  that  takes  into  account 
the  values  (information,  opinions,  and  preferences)  of  the  decision  maker  (Kirkwood, 
1997).  One  particularly  effective  method  of  creating  this  structured  tool  is  through  the 
use  of  value-focused  multiple  objective  decision  analysis  or  Value  Focused  Thinking 
(VFT).  Since  VFT  provides  an  objective  view  of  what  is  often  a  subjective  decision,  it 
will  allow  a  hierarchical  structuring  of  the  values  that  the  Air  Force  holds  with  regard  to 
utilities  privatization  and  facilitate  an  evaluation  of  the  contractor  proposals  under 
consideration  based  on  that  value  structure.  Therefore,  the  objective  of  this  thesis  will  be 
to  develop  a  VFT  model  for  utilities  privatization  as  a  reliable,  objective,  repeatable,  and 
defendable  evaluation  tool  that  can  be  used  universally  by  Air  Force  bases  when  making 
source  selection  decisions. 

To  achieve  this  objective,  the  following  research  questions  will  be  addressed 
during  this  research: 

1 .  What  values  does  the  Air  Force  hold  regarding  utilities  privatization?  What 
measures  can  be  developed  and  used  to  accurately  represent  and  evaluate 
those  values? 
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2.  Which  values  affect  the  contract  award  decision  most?  Will  this  hold  true  at 
all  bases  and  throughout  the  process  at  each  base? 

3.  Will  one  set  of  values  hold  true  for  all  utility  systems  (i.e.,  electrical,  water, 
wastewater,  and  natural  gas)?  Can  the  value  hierarchy  be  held  constant  with 
changes  made  only  in  the  measurements,  value  functions,  or  weights  for  each 
system  evaluation? 

1.5  Methodology 

In  this  thesis  research,  Value  Focused  Thinking  will  be  used  in  its  “traditional” 
sense  to  evaluate  alternatives  (contractor  proposals)  by  using  a  value  hierarchy  to  aid  in 
making  a  decision  (source  selection).  For  this  specific  problem,  the  ten-step  VFT 
methodology  proposed  by  Shoviak  (2001)  will  be  used  to  develop  the  mathematical 
model  with  the  aid  of  the  Air  Force  Civil  Engineer  Support  Agency  utilities  privatization 
team.  Since  the  model  is  based  on  values  inherent  to  the  decision  maker,  validation  of 
the  model  is  accomplished  by  the  decision  maker  throughout  the  model  development 
process.  To  further  validate  the  model,  it  will  be  used  to  evaluate  a  notional  data  set  that 
is  randomly  generated  specifically  for  the  purposes  of  this  research.  Performing  post¬ 
analysis  on  the  model’s  recommended  actions  regarding  this  data  set  will  provide 
validation  and  insight  into  the  usefulness  of  the  model. 

1.6  Scope  and  limitations  of  Research 

The  intended  scope  of  this  research  was  to  create  a  standardized  Value  Focused 
Thinking  model  that  will  allow  the  systematic  and  objective  evaluation  of  proposals 
submitted  by  contractors  competing  for  award  of  utility  privatization  contracts.  The 
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model  will  provide  a  tool  to  evaluate  source  selection  decisions  for  electrical  utility 
systems  at  Air  Force  installations.  Further  research  may  have  to  be  performed  to  convert 
the  model  into  one  that  is  more  useful  for  other  utility  systems  (water,  wastewater,  and 
natural  gas). 

Through  research  completed  at  AFIT  and  by  the  AFCESA  Utilities  Privatization 
Team,  it  was  discovered  that  no  active,  reserve,  or  air  national  guard  base  has  privatized 
their  electrical  system  using  the  current  RFP  solicitation  process.  Furthennore,  at  the 
time  of  this  research  no  bases  were  in  the  process  of  privatizing  their  electrical  systems. 
Since  it  was  impossible  to  test  the  developed  model  against  actual  proposals,  it  was  tested 
against  notional  data  sets  (proposal  scores)  generated  using  a  random  number  generator. 

1.7  Document  Structure 

There  are  four  remaining  chapters  in  this  thesis.  Chapter  2,  Literature  Review, 
provides  an  in-depth  presentation  of  the  existing  knowledge  base  and  consensus  of 
thought  regarding  utilities  privatization  and  Value  Focused  Thinking.  The  chapter  will 
allow  the  reader  to  develop  a  basic  understanding  of  the  concepts  critical  to  this  research. 
Chapter  3,  Methodology,  will  focus  on  the  technique  of  developing  and  using  a  VFT 
model  to  evaluate  the  research  objective  specified  in  Chapter  1.  It  will  also  explain  how 
the  post- award  analysis  was  conducted  using  the  model’s  results  as  well  as  explain  the 
validation  process  inherent  to  VFT.  Chapter  4,  Data  Analysis,  will  review  the  results  of 
the  post-award  analysis  conducted  on  the  notional  data.  It  will  also  explain  the  impact  of 
this  analysis  on  the  model  itself.  Chapter  5,  Results  and  Conclusions,  will  summarize  the 
results  of  the  research  effort  and  present  the  resulting  conclusions.  It  will  also  present 
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recommendations  for  implementation  of  this  model  and  for  further  research  within  this 
field. 
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Chapter  2.  Literature  Review 


This  chapter  provides  an  in-depth  presentation  of  the  existing  knowledge  base  and 
consensus  of  thought  regarding  utilities  privatization  and  Value  Focused  Thinking.  The 
chapter  begins  by  defining  privatization  not  only  in  the  private  and  government  sector, 
but  also  in  tenns  of  this  research  effort.  This  chapter  also  includes  a  discussion  of  the 
reasons  privatization  is  being  used,  concerns  about  privatization,  and  appropriate 
circumstances  for  the  use  of  privatization.  The  history  of  utilities  privatization  in  the 
Department  of  Defense  (DoD)  and  Air  Force  is  also  reviewed,  along  with  pertinent 
policies  and  a  discussion  of  the  current  Air  Force  source  selection  process.  Finally, 

Value  Focused  Thinking  is  described  in  tenns  of  Decision  Support  Systems  and  models. 

2.1  Defining  Privatization 

Before  developing  a  valid  source  selection  model,  a  good  working  definition  of 
privatization  is  critical  to  understanding  this  research  effort.  Defining  privatization  is 
more  difficult  than  it  sounds,  as  the  term  has  been  used  to  describe  the  following  types  of 
typical  privatization  strategies  (Barnekov  and  Raffel,  1990): 

1 .  Shifting  the  public  provision  of  goods  or  services  to  private  hands  through 
contracting  out  or  issuing  vouchers  but  still  providing  public  financing. 

2.  Complete  withdrawal  of  public  agencies  from  specific  goods  or  services 
with  the  supposition  that  private  institutions  such  as  firms,  families,  or 
voluntary  organizations  will  supply  those  goods  or  services. 

3.  The  sale  of  public  assets. 

4.  Deregulation  (the  removal  of  governmental  controls  on  the  private 
provision  of  a  good  or  service). 
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In  fact,  although  privatization  is  often  referred  to  and  discussed  as  one  continuous 
process,  the  concept  is  best  defined  as  a  group  of  policies  and  techniques  that  are 
designed  and  initiated  to  promote  private  sector  involvement  in  the  administration  or 
financing  of  goods  and  services  that  are  traditionally  viewed  as  the  responsibility  of  the 
government  (Auger,  1999).  Table  1  shows  how  privatization  efforts  have  been  initiated 
by  state  governments  in  the  past. 


Table  1.  Methods  Used  in  Privatization  Activities  Reported  by  States 

(Auger,  1999) 


Contracting 

80.0% 

Grants 

5.7% 

Public-private  Partnerships 

4.3% 

Volunteerism 

1.8% 

Vouchers 

0.7% 

Privatization  can  also  be  looked  at  from  the  viewpoint  of  two  competing  tactics: 
(1)  using  the  assumed  inherent  efficiency  of  the  private  marketplace  to  improve  the 
dispersion  of  goods  or  services  or  (2)  reducing  or  ending  public  provision  and  support  for 
specific  goods  and  services  completely  (load  shedding)  (Bamekov  and  Raffel,  1990). 
Sundquist  (1984)  points  out  that  the  first  of  these  strategies  is  the  elimination  of  what 
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“may  be  the  easiest  part  -  the  doing.  The  conceiving,  planning,  goal-setting,  standard¬ 
setting,  performance-monitoring,  evaluating,  and  correcting  all  remain  within  the 
government.”  This  transference  tactic  accurately  describes  the  process  currently  used  by 
the  Air  Force  to  privatize  its  utility  systems.  The  second  strategy  occurs  when 
government  entities  remove  themselves  partially  or  completely  from  the  role  of 
purchaser,  planner,  goal-setter,  perfonnance  monitor,  etc.,  for  a  specific  good  or  service 
area  (Kolderie,  1986).  Some  supporters  of  privatization  call  this  the  “real”  privatization 
because  it  eliminates  “public  spending  coalitions”  and  political  support  for  growth  in 
government  roles  (Butler,  1985).  For  this  method  to  be  considered  privatization,  other 
proponents  argue  that  the  government  or  public  entity  must  ensure  that  a  private  sector 
replacement  will  be  available  and  that  the  original  good  or  service  is  not  simply  being 
eliminated  (Sundquist,  1984). 

As  stated  in  Chapter  1 ,  privatization  can  be  generally  defined  as  the  transfer  of 
government-held  systems  to  private  (commercial)  sector  organizations.  This  transfer  can 
be  complete  (e.g.,  production,  conveyance,  and  billing)  or  partial  (e.g.,  production  only). 
In  more  specific  terms  for  the  purposes  of  this  research  effort,  privatization  will  be 
considered  as  the  transfer  of  government-held  utility  systems  (i.e.,  electrical,  water, 
wastewater,  and  natural  gas)  to  private  (commercial)  sector  organizations  with  the 
financing  and  oversight  of  the  systems  remaining  the  responsibility  of  the  government 
(i.e.,  Air  Force). 
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2.2  Why  Privatization? 

Public  utilities  have  been  under-funded  and  under-managed  for  too  many  years 
and  have  grown  too  dilapidated  and  costly  for  governments  to  bring  back  up  to 
satisfactory  operational  status  under  normal  funding  scenarios  (Orwin,  1999;  de  Luca, 
1997).  At  the  same  time,  the  public  is  clamoring  for  a  less  prominent  role  from 
government  to  reduce  the  bloated  bureaucracies  that  have  become  commonplace  (Wallin, 
1997).  This  situation  is  causing  government  managers  at  every  level  (national,  state,  and 
local)  to  realize  that  they  must  implement  systems  that  not  only  cut  operational  and 
capital  costs,  but  also  maintain  service  levels  and  have  more  than  a  “surface 
attractiveness”  and  simple  short-term  advantages  (Auger,  1999).  Throughout  the  world, 
privatization  has  been  hailed  as  the  most  efficient  means  of  creating  market  solutions  to 
what  have  become  complacent,  ineffective,  and  expensive  publicly  held  utility  systems 
(Orwin,  1999;  de  Luca,  1997;  Plaff,  2001). 

Most  people  accept  the  potential  for  privatization  to  yield  significant  social 
improvements  and  benefits  by  utilizing  private  sector  financial  and  organizational 
resources  to  “supplement  or  replace”  publicly  provided  goods  and  services  (Prager  and 
Desai,  1996).  The  strong  belief  that  the  private  sector  is  “inherently  dynamic,  productive, 
and  dependable,”  convinces  many  decision  makers  that  the  private  sector  can  and  will 
provide  less  costly  service  at  a  higher  service  level  than  current  publicly  owned  utilities 
(Bamekov  and  Raffel,  1990).  Another  factor  leading  to  this  belief  is  the  high  capital  cost 
associated  with  bringing  existing  utility  systems  up  to  current  technological  standards. 
Often,  without  expenditures  in  the  millions  or  billions  of  dollars,  these  systems  stand  no 
chance  of  being  brought  up  to  proper  working  condition  (Orwin,  1999;  de  Luca,  1997). 
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Since  private  companies  often  have  more  options  for  financing  these  capital  investments, 
proponents  of  utilities  privatization  point  to  high  capital  costs  as  a  strong  incentive  for 
publicly  owned  systems  to  change  hands. 

Supporters  of  privatization  also  point  out  that  without  the  need  to  support  these 
inefficient  systems  financially,  which  often  operate  at  substantial  losses,  public 
(governmental)  expenditures  and  taxes  can  be  greatly  reduced.  Additionally,  revenues 
can  be  generated  through  the  sale  and  transfer  of  the  utility  systems  as  well  as  the 
infrastructure  and  assets  associated  with  the  systems.  Utilities  privatization  supporters 
are  also  quick  to  mention  that  the  operation  and  funding  of  publicly  held  utility  systems 
are  extremely  politicized.  This  faction  touts  privatization  as  a  means  to  both  depoliticize 
the  management  and  financing  of  these  systems  and  eliminate  the  power  of  public  sector 
unions  (Bamekov  and  Raffel,  1990). 

2.3  Concerns  with  Privatization 

Even  proponents  of  privatization,  however,  admit  that  public  administrations 
should  avoid  “knee-jerk  reactions”  in  claiming  that  privatization  is  always  more  effective, 
more  efficient,  and  less  costly  than  the  goods  and  services  being  provided  by  government 
entities  (Prager  and  Desai,  1996).  In  fact,  most  opponents  of  privatization  consider  the 
belief  in  long-term  advantages  of  privatization  as  an  extremely  naive  viewpoint,  arguing 
that  as  the  sole  or  major  purchaser  of  goods  and  services  from  many  of  the  companies 
involved  with  privatization  efforts,  public  entities  may  become  hostage  to  the 
inefficiencies  and  bureaucracies  of  these  “single  seller”  monopolies.  In  essence, 
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privatized  utility  systems  could  end  up  in  the  same  place  from  which  they  started,  with 
high  prices  and  low  service  levels  (Bailey,  1987). 

Another  argument  presented  against  privatization  is  the  specter  of  client 
“creaming”  whereby  private  corporations,  in  order  to  show  good  results,  tend  to  provide 
services  only  to  the  easiest,  most  predictable  customers,  while  ignoring  the  “difficult  and 
unprofitable”  ones  (Bamekov  and  Raffel,  1990;  Orwin,  1999).  Many  people  fear  that  this 
will  simply  enlarge  the  rift  between  the  well-off  and  the  poor  in  tenns  of  the  quality  of 
goods  and  services  available  (Barnekov  and  Raffel,  1990;  Orwin,  1999).  This  situation 
has  been  seen  in  many  of  the  South  American  countries  that  have  initiated  privatization 
efforts  (Orwin,  1999). 

Other  opponents  point  to  the  threat  that  privatization  poses  in  regard  to 
corruption.  Once  the  private  sector  becomes  involved  with  government  and  gains  the 
ability  to  influence  political  decisions,  exploit  contracts,  and  create  cost  overruns,  the 
potential  for  fraud,  bribery,  and  kickbacks  exists.  Americans  have  seen  this  occur  in  their 
defense  industry,  highway  construction  and  maintenance  industry,  and  their  medical  care 
system  (Barnekov  and  Raffel,  1990).  In  order  to  prevent  this  problem,  more  and  more 
funds  must  be  spent  to  monitor  and  regulate  the  corporations  in  the  privatization  process 
(Bamekov  and  Raffel,  1990).  Although  competition,  the  main  market  factor  that  should 
provide  success  in  privatization,  usually  provides  benefits  initially,  those  benefits  may  be 
short-lived  (Kohn,  1986).  This  is  especially  true  with  regard  to  the  long-term  contracts 
(10-50  years)  that  government  entities  typically  pursue  with  privatization  efforts  (Hodge, 
1999). 
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Opponents  also  argue  that  one  of  the  most  worrisome  problems  with  privatization 
from  the  social  standpoint  is  that  although  cost  savings  and  benefits  may  be  realized  at 
the  organizational  level,  other  concerns  are  created  at  higher  levels.  These  opponents 
state  that  privatizing  a  good  or  service  results  in  lower  wage  levels,  greater  use  of  part- 
time  workers  with  fewer  fringe  benefits,  and  higher  unemployment  numbers.  These  all 
create  a  social  problem  that  drains  from  the  greater  good  and  imposes  monetary  and  life 
problems  on  public  employees  (Barnekov  and  Raffel,  1990). 

This  concern  with  the  social  impacts  of  privatization  is  one  of  the  larger  parts  of 
the  argument  against  it.  While  proponents  point  to  the  economic  good  resulting  from 
privatization,  opponents  focus  on  other  values,  such  as  “accountability,  equity,  service 
quality,  and  governmental  capacity.”  Although  competition  in  the  private  marketplace 
provides  obvious  potential  benefits,  it  also  creates  a  situation  in  which  private  firms  have 
an  agenda  that  does  not  necessarily  amount  to  creating  the  best  social  environment 
possible.  While  the  government  has  a  social  responsibility  to  provide  for  its  citizens  and 
conduct  itself  in  an  open  and  fair  manner,  private  corporations  are  not  held  to  the  same 
standard  (Starr,  1987). 

2.4  When  Privatization  Should  Be  Used 

All  of  the  arguments  against  privatization  indicate  that  a  well-defined  and 
effective  system  for  awarding,  implementing,  monitoring,  and  evaluating  a  privatization 
effort  is  fundamental  to  the  success  of  the  endeavor.  These  arguments  also  illustrate  that 
the  initial  decision  on  whether  or  not  to  privatize  must  be  based  on  a  solid  evaluation 
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process.  However,  much  of  the  current  argument  on  whether  or  not  to  privatize  a  system 
revolves  only  around  cost  effectiveness. 

A  great  deal  of  this  emphasis  on  cost  is  driven  by  the  relative  ease  of  obtaining 
and  comparing  cost  data,  especially  when  compared  to  the  often  difficult  task  of 
obtaining  service  level  data  (Bamekov  and  Raffel,  1990).  Prager  and  Desai  (1996)  state 
that  one  of  the  key  points  that  emerged  from  their  study  on  privatizing  local  government 
operations  was  that  making  the  right  decision  on  privatizing  or  not  requires  “an  approach 
that  focuses  on  and  appropriately  interprets  private  sector  -  public  sector  cost- 
effectiveness  comparisons”  (Prager  and  Desai,  1996).  They  also  state  that  although  many 
organizations  evaluating  the  cost-effectiveness  of  privatization  ignore  cost  tradeoffs,  the 
Office  of  Management  and  Budget’s  Circular  A-76  is  “as  complete  and  pragmatic  a 
document  as  is  likely  to  appear”  for  comparing  costs.  Table  2  illustrates  the  complexity 
of  the  cost  comparison  perfonned  with  the  A-76  process. 

The  A-76  process  applies  to  work  that  is  a  government  function  but  is  “contracted 
out”  for  accomplishment  by  a  private  entity.  However,  Prager  and  Desai  (1996)  also 
argue  that  the  A-76  process  is  flawed  because  it  bases  the  award  decision  on  low-cost 
alone.  This  lack  of  consideration  for  other  areas  such  as  efficiency  or  productivity  can 
cause  “woefully  inappropriate  policy  decisions”  (Prager  and  Desai,  1996).  These  same 
concerns  can  be  transferred  directly  to  most  privatization  efforts,  including  utilities 
privatization,  and  highlight  the  need  for  a  source  selection  tool  that  examines  tradeoffs 
rather  than  focusing  on  cost  alone. 
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Table  2.  In-House  Versus  Contract  Performance  Cost  Calculations 
(Pragerand  Desai,  1996) 


In-House  Performance  Costs 

Contractor  Perfonnance  Costs 

Personnel 

Contract  Price 

Materials  and  Supplies 

Contract  Administration 

Other  Specifically  Attributable  Costs 

Additional  Costs 

Overhead 

On-Time  Contract  Conversion  Costs 

Capital 

Gain  (Loss)  on  Disposal  or  Transfer  of 
Assets 

One-time  Conversion  Costs 

Federal  Income  Tax  Deduction 

Additional  Costs 

Total 

Total 

Most  of  the  emphasis  for  privatization  revolves  around  a  reduction  in  cost  with  no 
related  reduction  in  service  levels.  This  focus  dictates  that  proper  baseline  measurements 
must  be  taken  and  that  effective  comparison  areas  are  imperative.  The  A-76  Circular 
states  that  the  “contracting  out  process  cannot  get  off  the  ground  without  a  clear 
Performance  Work  Statement”  (Prager  and  Desai,  1996).  Similarly,  many  studies  show 
that  the  best  candidates  for  privatization  are  government  functions  that  have  identifiable 
and  specific  “service  tasks  and  performance  expectations  (Auger,  1999;  Barnekov  and 
Raffel,  1990;  Dilger,  Moffitt,  and  West,  1997;  and  Wallin,  1997).  Table  3  lists  various 
criteria  that  may  rule  out  privatization  as  an  option. 
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Table  3.  Conditions  and  Criteria  That  May  Dictate  Against  Contracting  Out 

(Wallin,  1997) 


1 .  When  other  privatization  strategies  are  deemed  to  be  more  effective 

2.  When  services  cannot  be  effectively  measured  as  to  cost,  quality,  process, 
and  outcomes 

3.  When  done  in  conjunction  with  services  cuts 

4.  When  privatization  is  explicitly  forbidden  by  existing  collective  bargaining 
agreements,  or  when  costs  outweigh  benefits 

5.  For  so-called  “core”  functions  of  government,  e.g.,  policy-making  or 
enforcement  functions 

6.  When  public  ends  (i.e.,  equity  access,  antidiscrimination)  are  ill-served  by 
private  provision 

7.  When  services  are  not  readily  available  from  the  private  sector 

8.  Where  legal  barriers  exist 


Some  government  services,  such  as  human  services  and  education,  have  not  been 
investigated  as  heavily  for  privatization  because  the  results  are  difficult  to  quantify. 
However,  other  government  services  have  been  more  aggressively  targeted  for 
privatization  because  of  the  ease  in  identifying  effective  measurement  devices.  Table  4 
shows  the  ten  most  privatized  services  in  America  in  1995.  The  privatization  of  utilities 
systems  would  appear  to  fit  into  this  group  exceptionally  well. 
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Table  4.  Ten  Most  Privatized  Services  in  1995 
(Dilger,  Moffitt,  and  West,  1997) 


1.  Vehicle  Towing/Service 

2.  Solid  Waste  Collection 

3.  Building  Security 

4.  Street  Repair 

5.  Ambulance  Services 

6.  Printing  Services 

7.  Street  Lighting/Signals 

8.  Drug/Alcohol  Treatment 

9.  Employment  and  Training 

10.  Legal  Services 


2.5  Utilities  Privatization  Within  the  DOD 

The  current  Department  of  Defense  vision  for  their  utility  systems  centers  on 
providing  or  acquiring  adequate  funding  and  management  to  bring  all  systems  up  to 
standards  by  2010.  In  order  to  incorporate  private  sector  innovations  and  efficiencies,  the 
DoD  prefers  utilities  privatization  as  their  method  of  modernization  and  sustainment 
(Marrs,  2003).  The  DoD,  driven  by  the  current  state  of  world  affairs  (Operations 
ENDURING  FREEDOM,  IRAQI  FREEDOM,  etc.),  is  also  trying  to  realign  itself  around 
its  core  competencies:  Air  and  Space  Superiority,  Global  Attack,  Rapid  Global  Mobility, 
Precision  Engagement,  Information  Superiority,  and  Agile  Combat  Support.  Many  DoD 
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leaders  feel  that  maintaining  and  operating  utility  systems  does  not  fall  within  any  of 
these  core  competencies.  Once  again,  privatization  has  been  targeted  as  the  preferred 
method  of  retaining  quality,  affordable  service  while  reducing  costs  (Marrs,  2003). 

This  vision  began  to  take  shape  in  December  of  1997  when  Defense  Reform 
Initiative  Directive  #9  (DRID  #9),  Privatizing  Utility  Systems,  was  issued  to  the  military 
services  (Army,  Navy,  and  Air  Force).  DRID  #9  directed  the  services  to  develop  and 
implement  plans  for  privatizing  all  of  their  utility  systems  (defined  as  electric,  water, 
waste  water,  and  natural  gas)  by  January  1,  2000,  except  where  privatization  of  a  system 
was  either  not  economically  feasible  or  not  appropriate  due  to  unique  security  reasons. 
DRID  #9  also  directed  the  establishment  of  “uniform  criteria  for  the  military  departments 
to  apply”  in  determining  whether  or  not  it  was  feasible  to  privatize  a  system.  The 
services  were  also  instructed  to  establish  a  timetable  with  benchmark  milestones  to  meet 
no  later  than  13  March,  1998  (Deputy  Secretary  of  Defense,  1997). 

In  January  of  1998,  DRID  #21,  Formation  of  the  Defense  Energy  Support  Center 
(DESC),  was  put  into  effect.  This  initiative,  along  with  redesignating  the  Defense  Fuel 
Supply  Center  as  the  DESC,  directed  the  DESC  to  develop  plans  to  execute  regional 
demonstrations  of  total  energy  management  and  consolidate  the  DoD’s  regional  energy 
efforts.  Part  of  this  responsibility  was  to  assist  the  services  with  privatizing  the  utility- 
related  infrastructure  that  was  initially  associated  with  the  regional  demonstrations 
(Deputy  Secretary  of  Defense,  1998). 

By  December  of  1998,  DRID  #  49,  Privatizing  Utility  Systems,  was  issued  as  the 
DoD  leadership  realized  that  the  number  of  systems  to  be  privatized  and  the  complexity 
of  the  privatization  process  made  the  goals  outlined  by  DRID  #  9  unrealistic. 
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Accordingly,  DRID  #49  extended  the  completion  goal  to  September  30,  2003.  It  also 
directed  that  the  services  observe  two  milestone  dates:  a  “Go/No-Go”  decision  for  all 
utility  systems  by  September  30,  2000,  regarding  whether  or  not  a  system  was  available 
for  privatization  (i.e.,  no  unique  security  requirements  and  economically  feasible),  and  a 
release  of  solicitations  for  all  systems  identified  as  available  by  September  30,  2001 
(Deputy  Secretary  of  Defense,  1998).  DRID  #49  also  required  the  military  services  to 
submit  their  revised  privatization  plans  by  December  23,  1998.  These  plans  were  to 
include  an  inventory  of  all  current  utility  systems,  including  those  that  were  on  the 
exemption  list.  The  plans  were  to  also  include  a  detailed  timeline  for  each  system 
showing  four  key  milestones:  Notice  of  Intent,  Study  Complete,  Solicitation,  and 
Contract  Award.  A  stipulation  was  also  included  that  called  for  the  services  to  submit 
quarterly  reports  summarizing  each  system’s  progress,  problems  encountered,  and  ways 
to  ensure  efficiency  and  eliminate  barriers  within  the  privatization  process  (Deputy 
Secretary  of  Defense,  1998).  It  also  set  forth  written  guidance  on  exempting  systems 
from  privatization,  competing  contracts,  and  conducting  economic  analyses.  Finally, 
DRID  #49  also  directed  action  be  taken  to  determine  the  amount  of  legislative  relief 
required  on  two  particular  obstacles:  the  10-year  limitation  on  utility  service  contracts 
and  the  tax  treatment  of  utility  system  conveyances  (Deputy  Secretary  of  Defense,  1998). 

The  DoD  leadership  soon  realized  that  the  time  requirements  outlined  in  DRID 
#49  were  also  too  restrictive.  Over  1,300  utility  systems  were  under  solicitation  or 
pending  the  release  of  a  Request  for  Proposal  (RFP)  in  September  of  2001.  The  utility 
industry  notified  the  DoD  that  this  was  saturating  the  market  and  decreasing  effective 
competition  (thus  eliminating  many  of  the  economic  advantages  of  privatization).  The 
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military  departments  concurred  with  the  utility  industry  report,  causing  the  DoD  to 
release  revised  guidance  on  utilities  privatization  in  October  of  2002  (Deputy  Secretary 
of  Defense,  2002). 

The  new  guidance  covered  all  aspects  of  the  privatization  process  and  directed  the 
services  to  complete  a  utilities  privatization  (UP)  evaluation  on  every  system  in  the  DoD 
inventory  by  September  30,  2005;  the  only  exceptions  were  utility  systems  designated  for 
closure  under  a  base  closure  law.  It  also  required  the  services  to  submit  a  revised 
privatization  plan  and  schedule  for  all  systems  by  October  23,  2002  (two  weeks  after 
receipt  of  the  memo).  Thereafter,  the  military  departments  were  to  submit  an  annual 
update  and  quarterly  status  reports  showing  a  comparison  of  forecast  timelines  to  actual 
timelines  (Deputy  Secretary  of  Defense,  2002). 

The  revised  guidance  also  included  three  interim  milestones:  the  establishment  of 
final  internal  milestones  for  each  service  by  October  15,  2002;  the  closure  of  RFPs  or 
submittal  of  certificates  of  exemption  for  80  percent  of  the  systems  specified  as  available 
for  privatization  within  each  component  by  September  30,  2003;  and  finalization  of 
Source  Selection  Authority  (SSA)  decisions  or  submittal  of  certificates  of  exemption  for 
65  percent  of  the  systems  specified  as  available  for  privatization  within  each  component 
by  September  30,  2004  (Deputy  Secretary  of  Defense,  2002).  The  revised  guidance 
represented  a  move  away  from  strict  deadlines  and  refocused  efforts  towards  interim 
milestones.  This  shift  appeared  to  show  recognition  by  the  DoD  leadership  that  a  more 
thorough  and  efficient  privatization  effort  could  be  realized  if  the  process  was  not  rushed. 
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2.6  Utilities  Privatization  Within  the  Air  Force 


In  the  meantime,  the  United  States  Air  Force  (USAF)  leadership  struggled  to  meet 
all  of  the  imposed  deadlines  and  adjust  accordingly  to  each  new  piece  of  guidance.  The 
first  milestone  the  Air  Force  met  was  the  requirement  outlined  in  DRID  #49  for  a 
“Go/No-Go”  decision  on  all  utility  systems  by  September  30,  2000.  The  Air  Force 
concluded  that  513  systems  were  suitable  for  privatization;  434  of  these  systems  were  to 
be  considered  “Go”  and  79  were  to  be  considered  “No  Go”  (Secretary  of  the  Air  Force, 
2000). 

In  response  to  the  “Revised  Utilities  Privatization  (UP)  Program  Guidance” 
Package  released  on  October  9,  2002,  the  USAF  released  internal  guidance  on  October 
23,  2002,  to  help  its  personnel  meet  the  revised  Utilities  Privatization  (UP)  guidance 
provided  by  the  DoD.  It  covered  the  UP  policies,  the  roles  and  responsibilities  of  players 
in  the  process,  and  the  entirety  of  the  process  up  to  solicitation  and  award  of  the  contract 
and  property  transfer.  The  Air  Force  also  established  the  Air  Force  Utility  Privatization 
Executive  Steering  Group  in  May  of  2003  to  provide  guidance  on  and  oversee  execution 
of  the  Air  Force’s  utility  privatization  program.  The  steering  group  was  tasked  with  three 
specific  functions:  advocating  the  use  of  USAF  and  DoD  resources  for  funding  of 
privatization  efforts,  monitoring  and  providing  necessary  support  to  integrated  process 
teams  and  other  teams  in  executing  actual  projects,  and  reviewing  and  coordinating  on 
UP  progress  reports  (Department  of  the  Air  Force,  2002). 
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2.7  Current  Source  Selection  Process 


To  meet  the  interim  goal  of  releasing  Requests  for  Proposal  (RFPs)  for  all 
applicable  utility  systems  in  the  inventory  by  September  30,  2001,  the  Air  Force  released 
an  RFP  template  on  February  11,  2000.  After  its  initial  release,  the  template  was 
amended  in  June  of  2000  and  again  in  February  of  2003.  Released  by  the  Air  Force  Civil 
Engineer  Support  Agency  (AFCESA),  the  most  recent  revision  of  the  template  provides  a 
comprehensive  process  for  preparing  RFPs.  It  describes  how  a  contractor’s  price 
schedule  should  be  based  on  inputs  such  as  the  purchase  price  of  the  system,  the  cost  of 
initial  capital  upgrades,  transition  costs,  etc.  It  also  covers  particular  descriptions, 
specifications,  and  work  statements  that  should  be  included  in  the  RFP  to  encompass  a 
wide  range  of  subjects  including:  service  requirements  and  performance  standards, 
employees,  security,  rights  of  way,  service  interruption  and  contingencies,  work  order 
response  requirements,  work  coordination  requirements,  environmental  compliance, 
upgrades,  renewals  and  replacements,  transition  plans,  and  specialty  training 
requirements.  The  template  also  specifies  which  clauses  from  various  regulations  are 
considered  mandatory  items  and  identifies  specific  contract  terms  and  contract 
administration  requirements.  The  RFP  template  also  lists  the  various  representations, 
certifications,  and  statements  required  of  the  contractors.  Finally,  the  template  includes  a 
detailed  section  on  how  a  contractor  should  prepare  its  proposal  (AFCESA,  2003). 

One  of  the  improvements  included  in  the  2003  revision  is  the  inclusion  of 
instructions  to  the  contractors  on  how  proposals  will  be  evaluated.  Shown  in  Figure  1, 
the  evaluation  process  is  very  detailed;  however,  the  instructions  do  not  specify  the 
criteria  to  be  used  in  evaluating  proposals.  Therefore,  this  research  is  focused  on  creating 
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an  objective,  repeatable  methodology  for  comparing  and  identifying  competitive 
proposals,  establishing  areas  for  clarifications  or  discussions,  and/or  differentiating 
between  proposals  for  final  award. 
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Figure  1.  Overall  Evaluation  Process  Outlined  in  RFP  Template 

(DESC,  2002:75) 


During  the  source  selection  process,  the  current  evaluation  tool  evaluates 
proposals  against  five  factors:  Mission  Capability,  Past  Performance,  Proposal  Risk, 
Socioeconomic  Plan,  and  Price.  During  the  evaluation,  Mission  Capability,  Past 
Performance,  and  Proposal  Risk  are  considered  the  most  important  factors  and  are 
accorded  equal  importance.  The  Socioeconomic  Plan  is  ranked  as  the  least  important  of 
the  five  factors.  Furthermore,  the  combined  non-cost  factors  are  considered  to  be 
significantly  more  important  than  price  (DESC,  2002). 

Mission  Capability  and  Proposal  Risk  are  each  broken  down  further  into  five  sub¬ 
factors:  Service  Interruption/Contingency  Plan,  Quality  Management  Plan,  Capital 
Upgrades  and  Renewals  and  Replacements  Plan,  Operational  Transition  Plan,  and 
Financial  Capability.  The  Service  Interruption/Contingency  Plan  and  Quality 
Management  Plan  are  considered  the  most  important  sub-factors  and  are  accorded  equal 
importance.  The  remaining  three  sub-factors  are  equally  important;  however,  each  is  also 
considered  less  important  than  the  Service  Interruption/Contingency  Plan  and  Quality 
Management  Plan  sub-factors  (DESC,  2002). 

Each  of  the  four  factors,  other  than  Price,  is  evaluated  against  a  scale  constructed 
specifically  for  that  particular  factor  and  receives  an  appropriate  rating.  The  ratings  and 
definitions  for  these  factors  are  defined  in  Tables  5  through  8.  The  evaluation  itself  is 
accomplished  by  four  sub-teams:  Technical/Cost  Realism  Evaluation  Team,  Past 
Performance  Team,  Price  Analysis  Team,  and  the  Socioeconomic  Team.  These  teams 
fall  under  the  auspices  of  the  Source  Selection  Evaluation  Team  (SSET),  which  is 
responsible  for  advising  the  Source  Selection  Authority  (SSA)  and  Source  Selection 
Authority  Board  (SSA  Board)  as  appropriate.  The  SSET  is  also  responsible  for 
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performing  the  comparative  analysis  of  proposals  received  as  well  as  conducting  the 
evaluation  in  an  impartial  and  equitable  manner.  The  evaluation  considers  tradeoffs 
among  price  and  non-price  factors  by  applying  weights  to  each  of  the  factors.  However, 
price  is  not  included  in  the  weighting;  instead,  it  is  simply  evaluated  for  reasonableness 
and  realism,  whether  it  represents  a  clear  understanding  of  the  requirements,  and 
accurately  reflects  the  market  conditions  and  competition.  Price  is  also  compared  against 
the  government’s  estimated  “Should  Cost”  of  operating  the  utility  system,  which  is  the 
amount  that  the  government  should  have  been  spending  to  maintain  and  update  the 
system  to  keep  it  at  or  above  industry  standards.  This  is  different  than  the  actual  amount 
that  the  government  was  spending  on  the  system.  The  difference  between  the  “should” 
and  actual  costs  represents  the  under-funding  that  caused  the  deficiencies  the  government 
is  trying  to  overcome  with  privatization.  As  long  as  the  offeror’s  proposed  price  is  lower 
than  the  government’s  “Should  Cost”,  a  proposal  is  considered  for  further  evaluation. 


Table  5.  Mission  Capability  Ratings 
(DESC,  2002:86) 


Rating 

Definition 

Exceptional  (Blue) 

Exceeds  specified  minimum  performance  or  capability 
requirements  in  a  way  beneficial  to  the  Government 

Acceptable  (Green) 

Meets  specified  minimum  performance  or  capability 
requirements  necessary  for  acceptable  contract  performance 

Marginal  (Yellow) 

Does  not  meet  some  specified  minimum  performance  or 
capability  requirements  necessary  for  acceptable  contract 
perfonnance,  but  any  proposal  inadequacies  are  correctable 

Unacceptable  (Red) 

Fails  to  meet  specified  minimum  perfonnance  or  capability 
requirements.  Proposal  with  an  unacceptable  rating  are  not 
awardable 
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Table  6.  Past  Performance  Ratings 
(DESC,  2002:115) 


Rating 

Definition 

Exceptional  /  High 
Confidence 

Based  on  the  Offeror’s  performance  record,  essentially  no 
doubt  exists  that  the  Offeror  will  successfully  perform  the 
required  effort. 

Very  Good  / 

Significant  Confidence 

Based  on  the  Offeror’s  perfonnance  record,  little  doubt 
exists  that  the  Offeror  will  successfully  perfonn  the  required 
effort. 

Satisfactory  / 

Confidence 

Based  on  the  Offeror’s  performance  record,  some  doubt 
exists  that  the  Offeror  will  successfully  perfonn  the  required 
effort. 

Marginal  /  Little 
Confidence 

Based  on  the  Offeror’s  performance  record,  substantial  doubt 
exists  that  the  Offeror  will  successfully  perform  the  required 
effort.  Changes  to  the  Offeror’s  existing  processes  may  be 
necessary  to  achieve  contract  requirements. 

Unsatisfactory  /  No 
Confidence 

Based  on  the  Offeror’s  performance  record,  extreme  doubt 
exists  that  the  Offeror  will  successfully  perfonn  the  required 
effort. 

Neutral  /  Unknown 
Confidence 

No  performance  record  identifiable  (see  FAR 

15.305[al[21[iiil  and  [iv|). 
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Table  7.  Proposal  Risk  Ratings 
(DESC,  2002:87) 


Rating 

Definition 

High  Risk 

Likely  to  cause  significant  disruption  of  schedule,  increased 
cost,  or  degradation  of  performance.  Risk  may  be 
unacceptable  even  with  special  contractor  emphasis  and 
close  government  monitoring. 

Moderate  Risk 

Can  potentially  cause  some  disruption  of  schedule,  increased 
cost,  or  degradation  of  performance.  However,  special 
contractor  emphasis  and  close  government  monitoring  will 
probably  be  able  to  overcome  difficulties. 

Low  Risk 

Has  little  potential  to  cause  disruption  of  schedule,  increased 
cost,  or  degradation  of  performance.  Normal  contractor 
effort  and  nonnal  government  monitoring  will  probably  be 
able  to  overcome  difficulties. 

Table  8.  Socioeconomic  Plan  Ratings 
(DESC,  2002:116) 


Rating 

Definition 

Excellent 

Offeror’s  plan  addresses  all  seven  items  required  by  the 
solicitation ,  OR 

Offeror  is  a  large  business  and  has  submitted  a 
socioeconomic  statement  as  well  as  an  approved 
subcontracting  plan  that  sufficiently  addresses  all  seven 
items  required  by  the  solicitation,  OR 

Offeror  is  a  small  business 

Very  Good 

Offeror’s  proposal  contains  at  least  five  items  required  by  the 
solicitation,  OR 

Offeror  is  a  large  business  and  has  submitted  an  approved 
subcontracting  plan 

Good 

Offeror’s  proposal  contains  at  least  three  items  required  by 
the  solicitation,  OR 

Offeror’s  method  of  securing  gas  supplies  results  in  there 
being  no  subcontracting  opportunities  under  this  proposal 
(e.g.,  offeror  owns  its  own  production  and/or  has  pre-existing 
transportation  agreements) 

Poor 

Offeror  fails  to  submit  infonnation  that  satisfies  any  of  the 
requirements  listed  above  (applicable  to  large  businesses) 
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2.8  A  Distinct  Need  for  a  Definitive  Air  Force  Source  Selection  Process 

Awarding  contracts  in  the  utilities  privatization  arena  requires  a  decision  that 
considers  tradeoffs  between  multiple  competing  objectives  such  as  economic  feasibility, 
security  requirements,  readiness  requirements,  reliability,  service,  and  of  course  cost.  As 
with  most  decisions  involving  tradeoffs,  it  is  helpful  to  have  a  structured,  standardized 
technique.  The  methodology  outlined  in  the  current  RFP  template  is  a  step  in  the  right 
direction;  however  it  has  two  serious  flaws:  it  does  not  eliminate  subjectivity  and  it  is  not 
based  on  a  quantitative  comparison  of  the  proposals. 

Although  the  evaluation  method  currently  in  use  attempts  to  eliminate  subjectivity 
by  creating  scales  upon  which  four  of  the  evaluation  factors  are  “graded,”  three  of  the 
scales  are  not  based  upon  repeatable  scoring  techniques.  The  ratings  assigned  to  the 
Mission  Capability  Factor,  Proposal  Risk  Factor,  and  Past  Performance  Factor  are  based 
on  the  evaluation  team’s  opinions  regarding  the  appropriate  category  for  each  proposal. 
The  lack  of  specificity  within  each  of  the  category  definitions  for  these  three  factors 
might  result  in  the  team  choosing  the  best  of  several  bad  choices,  rather  than  realizing 
they  have  several  proposals  that  should  not  even  be  considered  without  resubmission. 
These  “soft”  definitions  make  it  difficult  for  the  teams  to  replicate  their  scoring  since  the 
evaluations  are  based  heavily  on  opinion  and  cross-comparison,  which  can  change  every 
time  the  proposals  are  evaluated. 

However,  the  more  critical  flaw  of  the  current  evaluation  process  is  the  fact  that 
there  is  no  quantitative  comparison  capability.  Instead,  the  ratings  assigned  to  the  various 
factors  are  compared  on  a  subjective  basis.  By  not  quantitatively  defining  weights  within 
the  model  and  instead  using  ambiguous  terms  such  as  most  important  and  less  important, 
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the  evaluation  process  becomes  subjective  at  the  highest  level  of  comparison.  Without 
this  quantifiable,  meaningful  comparison,  defensibility  of  the  evaluation  is  virtually 
eliminated. 

Another  sub-flaw  of  this  tool  is  that  by  evaluating  the  proposals  at  such  high 
levels  and  only  breaking  it  down  into  five  factors,  the  amount  of  information  that  must  be 
evaluated  against  each  of  the  already  subjective  scales  is  staggering.  A  much  more 
effective  and  meaningful  comparison  could  be  obtained  by  evaluating  at  the  lower  levels 
of  the  proposals.  The  current  methodology  attempts  this  by  breaking  mission  capability 
and  proposal  risk  into  subfactors  (Service  Interruption/Contingency  Plan,  Quality 
Management  Plan,  Capital  Upgrades  and  Renewals  and  Replacements  Plan,  Operational 
Transition  Plan,  and  Financial  Capability),  but  these  subfactors  defined  are  still  such 
enonnous  categories  to  evaluate  that  they  do  not  facilitate  making  the  evaluation 
objective. 

Despite  these  flaws,  the  privatization  of  a  utility  system  is  a  very  pennanent 
undertaking.  Contracts  are  usually  signed  for  time  periods  of  10  to  50  years.  Even  if  a 
contractor  was  to  fail  or  be  released  from  the  contract  for  poor  perfonnance,  it  would  be 
nearly  impossible  for  a  DoD  installation  to  reassume  responsibility  for  the  operation  of 
the  system.  Therefore,  the  permanency  of  the  privatization  decision  makes  it  essential  to 
have  good  decision  support  and  insight.  To  obtain  adequate  information  about  the  source 
selection  decision  being  made,  it  is  often  necessary  and  useful  to  have  more  than  one 
decision  evaluation  process.  One  of  the  most  common  lessons  learned  in  privatization 
efforts  around  the  world  is  that  a  reliable,  objective,  repeatable,  and  defendable  source 
selection  process  is  absolutely  necessary  (Orwin,  1999;  de  Luca,  1997).  Orwin  (1999) 
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states  that  not  only  is  a  standardized  source  selection  process  needed,  but  because  of  the 
enormous  potential  impact  of  these  contracts,  a  backup  (or  secondary)  source  selection 
tool  should  also  be  used  to  ensure  the  “accuracy”  of  the  primary  tool  (Orwin,  1999). 

2.9  Value  Focused  Thinking 

2.9.1  Decision  Support  Systems 

Decision  makers  evaluating  the  privatization  of  utility  systems  under  their  control 
are  faced  with  an  incredibly  difficult  and  complex  decision.  As  previously  discussed, 
awarding  contracts  in  the  utilities  privatization  arena  requires  a  decision  that  considers 
tradeoffs  between  multiple  competing  objectives.  The  priority  or  weight  that  individual 
tradeoffs  receive  can  become  very  subjective.  However,  a  decision  of  such  magnitude 
and  long  lasting  consequence  as  awarding  a  utilities  privatization  contract  needs  to  be 
assessed  as  objectively  as  possible.  One  thing  that  might  help  is  to  follow  a  structured 
decision  making  process  (Ragsdale,  2001).  By  following  a  structured  process,  such  as  a 
Decision  Support  System  (DSS),  the  decision  maker  is  much  more  likely  to  fully 
understand  the  decision  being  made  as  well  as  choose  the  best  alternative. 

2.9.2  Models 

Most  DSS  models  consist  of  three  basic  components:  data  collection,  a  model, 
and  presentation  (Post  and  Anderson,  2003).  The  most  important,  yet  most  difficult, 
component  is  the  creation  of  the  model.  However,  the  difficulty  of  modeling  the  problem 
is  far  outweighed  by  the  benefits  gained.  Models  represent  a  more  economical  means  of 
replicating  and  testing  decisions;  it  is  usually  more  feasible  to  implement  and  evaluate  a 
model  than  test  the  real-life  systems  using  a  trial-and-error  methodology.  Models  can 
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also  be  used  to  evaluate  literally  thousands  of  scenarios  and  can  be  tested  for  sensitivity; 
therefore,  they  offer  extremely  valuable  insight  and  understanding  into  the  decision 
(Ragsdale,  2001).  Models  also  help  eliminate  bias  and  common  errors  by  tempering  the 
tendency  of  decision  makers  to  “fly  by  the  seat  of  their  pants”  and  use  “rules  of  thumb” 
(Post  &  Anderson,  2003). 

Most  conventional  approaches  to  making  decisions  focus  on  alternatives.  These 
approaches  are  often  lumped  into  what  is  termed  Alternative  Focused  Thinking  (AFT). 
AFT  is  typically  what  most  decision  makers  use  when  confronted  with  a  problem;  it  is 
essentially  identifying  the  available  alternatives  and  evaluating  those  alternatives  based 
on  their  respective  merits  as  shown  in  Figure  2.  This  typically  results  in  the  evaluation  of 
alternatives  that  are  easily  identifiable  and  the  selection  of  the  “least  worst”  alternative. 
However,  these  alternatives  are  often  relevant  only  because  they  are  a  means  to  achieving 
the  decision  maker’s  values  (Keeney,  1993).  Keeney  argues  that  AFT  is  reactive,  not 
proactive,  and  puts  the  cart  of  alternative  identification  before  the  horse  of  value 
articulation  (Keeney,  1994). 
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Figure  2:  Alternative  and  Value  Focused  Thinking  Approaches 

(Katzer,  2002:27) 


Value  Focused  Thinking  (VFT),  however,  provides  a  structured,  standardized 
technique  that  takes  into  account  the  values  (information,  opinions,  and  preferences)  of 
the  decision  maker  (in  this  case  the  United  States  Air  Force)  (Kirkwood,  1997).  VFT 
allows  decision  makers  to  define  what  they  value  and  what  is  fundamentally  important  in 
a  specific  decision  situation  (Keeney,  1994).  These  values  should  be  the  driving  force  in 
the  decision  making  process.  They  allow  the  decision  maker  to  identify  alternatives  that 
more  completely  and  satisfactorily  match  their  fundamental  values. 

VFT  is  a  systematic  procedure  for  focusing  the  decision  maker  on  the  core 
activities  in  solving  a  decision  problem.  VFT  can  often  take  a  subjective,  imprecise,  and 
potentially  confusing  decision  problem  and  allow  the  user  to  objectively  weigh  the 
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problem  against  their  values  to  give  the  decision  maker  focused  insight  and  facilitate  a 
better  decision.  VFT  also  helps  the  decision  maker  discover  hidden  objectives,  leads  to 
more  productive  information  collection,  improves  communication  between  concerned 
parties,  facilitates  stakeholder  involvement,  and  enhances  the  “coordination  of 
interconnected  decisions”  (Keeney,  1994). 

By  employing  the  VFT  methodology,  it  is  possible  to  create  a  quantitative 
structuring  of  the  Air  Force’s  values  regarding  utilities  privatization  and  then  evaluate  the 
source  selection  decision  based  on  that  value  structure.  The  utilities  privatization  award 
decision  consists  of  competing  objectives  and  hard  to  define  goals  and  measures; 
however,  the  VFT  process  will  allow  these  hard  to  define  goals  and  measures  to  be 
converted  into  clear  objectives;  thereby  making  it  possible  to  determine  an  infonned, 
comprehensive,  and  defendable  decision  that  accounts  for  the  multiple  competing 
objectives  (Katzer,  2002).  For  this  research,  the  ten-step  VFT  methodology  outlined  by 
Shoviak  (2001)  was  used  to  develop  the  mathematical  model  with  a  representative  from 
the  Air  Force  Civil  Engineer  Support  Agency  utilities  privatization  team  serving  as  the 
primary  decision  maker  (PDM). 

2.9.3  VFT  Ten  Step  Process 

2. 9. 3.1  Step  1  -  Problem  Identification 

Although  seemingly  very  simple  and  straightforward,  this  step  can  be  deceiving 
since  it  is  often  assumed  that  the  fundamental  decision  being  considered  is  fully  known 
and  understood.  However,  this  step  is  by  far  the  most  critical  since  there  is  typically  only 
general  knowledge  available  regarding  the  problem.  Therefore,  it  is  imperative  that  the 
decision  maker  and  all  involved  in  the  process  properly  identify,  define,  and  understand 
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the  problem  being  addressed.  First  and  foremost,  it  is  extremely  important  the  decision 
maker  confirm  that  an  actual  decision  to  be  evaluated  exists. 

Proper  completion  of  this  step  allows  the  decision  maker  to  fully  understand  the 
decision  context,  objective,  and  possible  directions/preferences  of  the  stakeholders 
involved.  If  the  wrong  problem  is  addressed,  then  the  entire  resulting  effort  can  go  in  the 
wrong  direction  and  result  in  wasted  time  and  effort.  A  fully  defined  objective  question 
allows  everyone  involved  in  the  decision  process  to  fully  grasp  what  the  decision  entails. 
By  emphasizing  the  definition  of  the  problem,  the  decision  maker  can  help  ensure  that  the 
outcome  of  the  decision  analysis  is  ultimately  useful. 

2. 9.3.2  Step  2  -  Value  Hierarchy  Construction 

A  value  hierarchy  is  a  graphical  representation  of  the  decision  maker’s  values 
with  respect  to  the  decision  being  made.  Value  hierarchies  allow  the  decision  makers  and 
those  involved  in  the  decision  process  to  visualize  their  values  and  better  understand  how 
the  ensuing  value  system  influences  the  decision  process.  This  visualization  process 
allows  the  stakeholders  involved  to  not  only  solidify  their  values  but  also  to  evaluate  the 
thoroughness  of  the  value  set  depicted. 

Although  the  values  portrayed  in  the  value  hierarchy  are  detennined  by  the 
decision  maker  (or  a  chosen  proxy),  there  are  several  methods  to  facilitate  the  elicitation 
of  the  values.  Most  VFT  models  are  created  with  a  top-down  approach,  which  recognizes 
the  lack  of  clear  alternatives  and  allows  the  identification  of  additional  alternatives.  With 
this  approach,  the  main  objective  is  determined  and  then  the  values  related  to  this  main 
objective  are  identified  and  iteratively  decomposed  until  evaluation  measures  can  be 
developed.  The  focus  is  on  values  first  and  alternatives  second.  However,  in  some  cases, 
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the  alternatives  may  be  better  known  and  a  bottom-up  approach  may  be  the  best  way  to 
develop  the  value  hierarchy  (Kirkwood,  1997).  Using  this  approach,  the  alternatives  are 
evaluated  against  each  other  to  determine  their  differences  and  evaluation  measures  are 
developed  to  measure  these  differences.  The  measures  are  then  grouped  into  values  and 
the  values  are  grouped  to  form  layers  within  the  hierarchy  until  the  final  objective  is 
reached. 

Another  method  of  generating  values  is  called  the  “gold  standard”  and  consists  of 
a  review  of  relevant  literature  (Weir,  2003;  Kirkwood,  1997).  With  this  method,  the 
analyst  uses  documents  such  as  the  strategic  objectives,  vision,  and  plan  of  an 
organization;  regulations;  laws;  etc.,  to  generate  values  for  the  decision  maker’s 
consideration.  By  basing  values  on  published  source  documents,  the  analyst  is  able  to 
develop  a  hierarchy  that  is  much  more  defensible  to  scrutiny  from  either  the  decision 
maker  or  people  outside  of  the  decision  process.  Developing  a  “strawman”  hierarchy 
with  the  gold  standard  also  allows  the  analyst  to  facilitate  and  guide  the  rest  of  the 
hierarchy  construction  process;  the  “strawman”  serves  to  generate  a  higher  level  of 
efficiency  by  providing  an  initial  starting  point  that  can  be  reviewed  and  changed  by  the 
decision  maker  to  more  accurately  reflect  his  or  her  personal  hierarchy  (Katzer,  2002). 

The  “silver  standard,”  or  casual  empiricism,  is  another  method  for  developing 
values  (Weir,  2003;  Kirkwood,  1997).  The  silver  standard  involves  sessions  with  the 
relevant  stakeholders  that  are  often  conducted  in  large  groups.  There  may  only  be  a  small 
number  of  people  directly  interested  in  a  problem,  but  there  are  often  other  people  who 
have  valuable  experience  or  information  pertaining  to  the  problem;  consulting  these 
people  can  help  the  analyst  ensure  that  all  relevant  details  are  included  (Kirkwood,  1997). 
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This  method  often  produces  a  simpler,  more  logical  structure  than  the  gold  standard  that 
is  more  easily  understood. 

Yet  another  method  of  generating  values  is  the  “platinum  standard”  (Weir,  2003). 
Although  the  platinum  standard  focuses  on  interviews  with  key  leadership  and  technical 
personnel  involved  in  the  decision,  it  is  best  to  start  with  vision  statements,  strategic 
objectives,  and  other  “doctrinal  publications”  as  discussed  in  the  gold  standard.  The 
platinum  standard  can  actually  be  regarded  as  an  extension  or  improvement  of  the  gold 
standard.  This  method  will  usually  produce  the  most  insightful,  simple,  and  logical 
structure  for  the  value  hierarchy. 

Once  the  values  relevant  to  the  decision  being  made  are  generated,  they  are 
arranged  in  a  tree-like  structure  with  the  fundamental  objective  at  the  top.  Beneath  the 
fundamental  objective,  the  tiers  of  the  hierarchy  branch  out  to  represent  the  full  spectrum 
of  values.  Tiers  are  groupings  of  values  that  have  the  same  level  of  importance  in  the 
value  hierarchy.  Depending  on  how  complex  the  decision  and  value  set  associated  with  it 
are,  there  can  be  any  number  of  tiers  in  a  value  hierarchy. 

In  order  to  help  visualize  what  a  value  hierarchy  looks  like,  consider  the  example 
of  a  person  shopping  for  a  new  truck  (Figure  3).  The  fundamental  objective  for  this 
person  is  to  buy  the  best  truck.  The  first  tier  of  values  (cost,  performance,  and 
appearance)  represent  the  values  the  decision  maker  (shopper)  considers  the  most 
important  in  the  decision.  The  decision  maker  has  further  decomposed  performance  into 
a  second  tier  of  values  (power  and  off-road  capability). 


41 


Figure  3.  Example  Value  Hierarchy 


There  are  several  desired  properties  for  a  value  hierarchy:  completeness, 
nonredundancy,  independence,  operability,  and  “small”  size  (Kirkwood,  1997). 
Completeness  (i.e.,  collectively  exhaustive)  addresses  whether  or  not  the  hierarchy 
includes  all  of  the  evaluation  concerns  that  are  important  to  the  decision.  Completeness 
also  requires  that  the  evaluation  measures  adequately  and  appropriately  measure  how 
well  an  alternative  attains  the  objective  associated  with  each  measure.  If  the  hierarchy  is 
not  complete,  then  it  can  obviously  lead  to  false  insight  based  on  the  missing  information. 

Nonredundancy  (i.e.,  mutually  exclusive)  simply  means  that  a  particular  value  is 
not  covered  more  than  once  within  the  hierarchy.  In  other  words,  no  values  in  the 
hierarchy  overlap  (Kirkwood,  1997).  Nonredundancy  prevents  the  values  within  the 
hierarchy  from  being  given  more  weight  (i.e.,  importance)  than  the  decision  maker 
intends.  Similar  to  non-completeness,  redundant  values  can  contribute  to  false  insight 
being  gained  from  the  hierarchy. 
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Independence  (i.e.,  decomposability)  means  that  the  score  of  one  measure  within 
the  hierarchy  does  not  affect  the  score  of  any  other  measure.  For  instance,  in  the  truck 
buying  example,  suppose  the  measure  for  power  was  horsepower  and  the  measure  for  off¬ 
road  capability  was  acceleration.  These  two  measures  would  not  be  independent  since  a 
higher  score  for  horsepower  would  always  result  in  a  higher  score  for  acceleration  (all 
other  things  remaining  equal).  Therefore,  the  score  for  acceleration  is  dependent  on  the 
score  for  horsepower. 

Operability  deals  with  whether  or  not  the  people  using  the  model  understand  it 
(Kirkwood,  1997).  Although  the  hierarchy  must  be  technically  accurate  and  complete,  it 
must  also  be  understandable  by  the  target  audiences.  This  sometimes  requires 
compromises  on  some  of  the  other  characteristics  of  the  hierarchy  (Kirkwood,  1997). 

This  is  especially  important  if  the  decision  is  very  complex  but  not  all  of  the  target 
audience  is  an  expert  or  specialist  in  the  area  of  study. 

Finally,  small  size  addresses  the  issue  of  how  the  size  of  the  hierarchy  affects  the 
ability  to  explain,  understand,  and  use  the  model.  All  things  remaining  equal,  a  smaller 
hierarchy  can  be  communicated  more  easily  and  require  fewer  resources  for  evaluation 
purposes  (Kirkwood,  1997).  Unfortunately,  the  tendency  for  complex  projects  is  to  add 
values  and  measures  to  try  and  capture  every  possible  variable.  Consequently,  the  model 
becomes  so  large  and  cumbersome  that  understanding  it  or  evaluating  alternatives  against 
it  is  technically  or  monetarily  infeasible.  Kirkwood  (1997)  argues  that  the  quest  for 
completeness  and  detail  must  be  balanced  against  a  realistic  time  frame  and  budget. 
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2.933  Step  3  -  Evaluation  Measure  Development 

In  order  to  evaluate  alternatives  with  a  value  hierarchy,  it  is  necessary  to  create 
evaluation  measures  which  assess  the  degree  of  attainment  for  the  values.  These 
measures  allow  alternatives  to  receive  an  objective  and  unambiguous  rating  with  respect 
to  each  value  (Kirkwood,  1997).  As  seen  in  Figure  4,  it  is  possible  for  a  value  to  have 
more  than  one  measure  associated  with  it  in  order  to  accurately  capture  how  each 
alternative  impacts  that  particular  value.  For  instance,  off-road  capability  could  have 
multiple  evaluation  measures:  four-wheel  drive,  tire  size,  and  suspension  rating.  These 
multiple  measures  would  give  a  more  complete  picture  of  an  alternative’s  contributions 
towards  off-road  capability. 
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Figure  4.  Example  Value  Hierarchy  with  Measures 


Evaluation  measures  can  be  classified  into  differing  types  of  scales.  These  scales 
are  commonly  classified  as  natural  or  constructed  and  direct  or  proxy  (Kirkwood,  1997). 
A  natural  scale  is  one  that  is  in  general  use  and  has  a  common  interpretation  by  all  users 
(Weir,  2003).  An  example  of  a  measure  with  a  natural  scale  is  the  price  paid  measure 
under  the  cost  value  in  Figure  4.  A  constructed  scale  is  used  when  no  natural  scale  is 
available  and  is  developed  for  use  in  a  specific  decision  problem  to  measure  the  degree  of 
attainment  for  a  particular  objective  (Kirkwood,  1997).  Constructed  scales  are  often 
categorical  in  nature  and  define  a  finite  number  of  levels  of  attainment.  An  example  of  a 
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constructed  scale  in  the  truck  buying  hierarchy  is  the  suspension  measure  under  the  off¬ 
road  capability  value  where  the  categories  are  standard,  off-road,  and  towing.  The  score 
given  to  an  alternative  could  differ  from  decision  maker  to  decision  maker,  so  each 
category  has  to  be  defined  in  detail  (constructed)  to  eliminate  bias  and  confusion  during 
the  scoring  of  alternatives. 

The  scales  are  also  classified  as  either  a  direct  or  proxy  scale.  Direct  scales  focus 
on  attainment  of  the  objective  itself  and  directly  measure  the  level  of  attainment 
(Kirkwood,  1997).  The  most  commonly  used  direct  scale  (for  Americans)  is  dollars  ($) 
to  reflect  the  measure  of  cost.  Proxy  scales  focus  on  the  direct  measurement  of  an 
associated  objective,  which  is  selected  such  that  it  will  accurately  reflect  attainment  of  the 
primary  objective  (Kirkwood,  1997).  An  example  of  a  proxy  scale  is  the  use  of  the  Gross 
National  Product  (GNP)  to  reflect  a  measure  of  a  nation’s  economic  well  being  (Weir, 
2003). 

When  deciding  which  evaluation  measures  and  associated  scales  to  use,  the  order 
of  preference,  as  shown  in  Table  9,  is  typically  natural  direct,  constructed  direct,  natural 
proxy,  and  finally  constructed  proxy.  However,  several  other  things  need  to  be  kept  in 
mind  when  determining  scales.  Natural  scales  do  not  require  as  much  development  time 
and  are  less  controversial  because  they  are  in  general  use  (Kirkwood,  1997).  However, 
there  are  a  finite  number  of  natural  scales  and  the  data  to  be  measured  against  a  natural 
scale  is  not  always  available.  The  relative  importance  of  the  objective  being  measured 
must  be  taken  into  account  also  (Kirkwood,  1997).  Using  a  constructed  scale  that 
combines  several  natural  scales  may  more  accurately  reflect  the  value  of  a  particular 
objective  instead  of  several  measures  with  natural  scales  for  each  one.  The  type  of 
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audience  reviewing  the  results  also  figures  into  measure  development  (Kirkwood,  1997). 
Since  the  measures  and  scales  need  to  be  meaningful  to  the  decision  maker,  a  less 
technical  audience  may  require  measures  that  are  also  less  technical  and  more  operable  in 
nature.  Finally,  measures  must  be  able  to  pass  the  clairvoyance  test  (Kirkwood,  1997). 
Kirkwood  (1997)  asks,  “If  a  clairvoyant  were  available  who  could  foresee  the  future  with 
no  uncertainty,  would  this  clairvoyant  be  able  to  unambiguously  assign  a  score  to  the 
outcome  from  each  alternative?” 


Table  9.  Preferred  Order  of  Use  for  Evaluation  Measure  Scales 

(Weir,  2003) 


Natural 

Constructed 

Direct 

1 

2 

Proxy 

3 

4 

2.9. 3.4  Step  4  -  Value  Function  Creation 

To  use  the  VFT  process,  it  is  necessary  to  define  Single  Dimensional  Value 
Functions  (SDVFs)  to  allow  the  scales  defined  for  the  evaluation  measures  to  be 
converted  to  a  common  scale  so  that  scores  can  be  combined,  compared,  and  analyzed.  A 
common  scale  is  a  necessity  because  of  the  multiple  evaluation  measures  and  thus 
multiple  evaluation  units  that  exist  within  the  model.  Therefore,  SDVFs  convert  the 
individual  units  of  each  measure  into  “value”  units  that  most  often  range  from  zero 
(lowest  attainment)  to  one  (highest  attainment).  In  other  words,  the  least  preferred  score 
for  a  particular  measure  would  earn  a  “value”  of  zero  while  the  most  preferred  score 
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would  earn  a  “value”  of  one  (Kirkwood,  1997).  The  main  purpose  of  this  step  is  to 
convert  subjective  decisions  into  scores  that  can  be  examined  objectively  by  the  decision 
maker. 

Each  individual  SDVF  is  defined  by  its  shape,  which  is  detennined  by  soliciting 
input  from  the  decision  maker  or  subject  matter  experts  (SMEs)  to  detennine  how  the 
evaluation  measure  scores  should  be  converted  into  “value”  units.  There  is  one  major 
restriction  on  the  shape  of  the  SDVF.  It  must  be  monotonic  in  nature,  which  means  that 
the  slope  of  the  SDVF  is  either  positive  or  negative  across  the  entire  range  of  the 
function;  it  cannot  be  positive  and  negative  within  the  same  function.  In  other  words,  an 
increasing  function  indicates  that  the  higher  levels  of  the  evaluation  measure  are  always 
preferred  over  the  lower  levels.  Similarly,  a  decreasing  function  indicates  that  the  lower 
levels  of  an  evaluation  measure  are  always  preferred  over  the  higher  levels. 

The  shape  of  each  SDVF  is  determined  by  the  returns  to  scale  associated  with  the 
respective  evaluation  measure.  Returns  to  scale  can  be  broken  down  into  four  categories: 
constant,  increasing,  decreasing,  or  a  combination  of  the  above.  Kirkwood  (1997)  states 
there  are  two  possible  shapes  to  reflect  these  returns  to  scale:  piecewise  linear  and 
exponential.  These  shapes  are  all  represented  in  Table  10.  Piecewise  linear  functions  are 
typically  used  when  there  are  a  finite  number  of  scoring  levels  for  the  measure,  such  as 
with  a  categorical  measure  (Kirkwood,  1997).  Exponential  functions,  on  the  other  hand, 
are  used  when  a  measure  has  an  infinite  number  of  scoring  levels.  For  the  truck  buying 
example,  Figures  5  and  6  show  a  monotonically  increasing  piecewise  linear  function  and 
a  monotonically  decreasing  exponential  function  for  the  color  and  cost  evaluation 
measures,  respectively. 
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Table  10.  Common  SDVF  Shapes 


Piecewise  Linear 

Exponential 

Constant 

_ 1 

Increasing 

A 

Decreasing 

1 _ 

Other 

1 

_ 1 
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Figure  6.  Monotonically  Decreasing  Exponential  SDVF 
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2.93.5  Step  5  -  Value  Hierarchy  Weighting 

After  the  values  of  the  decision  maker  are  clearly  identified,  and  the  related 
evaluation  measures  are  created,  it  is  important  to  detennine  the  relative  importance  of 
the  values  and  measures.  Weighting  the  value  hierarchy  allows  the  PDM  (or  proxy)  to 
truly  define  his  or  her  values  by  indicating  which  values  are  of  the  highest  importance. 
Weights  can  be  assigned  locally  or  globally  depending  on  which  method  the  analyst  feels 
better  represents  the  decision  maker’s  value  ranking. 

2.9.3.5.1  Local  Weighting 

Local  weighting  refers  to  the  relative  importance  assigned  to  values  or  measures 
that  are  located  on  the  same  tier  and  within  the  same  branch  of  the  hierarchy  (Weir, 
2003).  The  sum  of  the  local  weights  (same  tier  of  any  given  branch)  must  sum  to  one  as 
seen  in  Figure  7.  Local  weighting  is  usually,  but  not  always,  accomplished  with  a  top- 
down  approach  typically  using  either  the  direct  weighting  or  swing  weighting  technique. 
The  direct  weighting  method  is  often  referred  to  as  the  100-marble  weighting  system. 
With  this  method,  the  PDM  is  given  100  imaginary  marbles  and  asked  to  apportion  the 
marbles  in  imaginary  boxes  to  signify  the  importance  of  the  values  (or  measures)  on  the 
same  tier  within  a  branch  of  the  hierarchy.  The  number  of  marbles  in  each  box  (divided 
by  100)  thus  represents  the  PDM’s  importance  ranking  for  each  value  (or  measure).  This 
technique  gives  the  decision  maker  a  much  more  direct,  visual,  and  clear  understanding 
of  the  importance  each  weight  represents. 
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Figure  7.  Example  Hierarchy  with  Local  Weights 

Swing  weighting  can  actually  be  accomplished  with  two  slightly  different 
techniques.  The  first  requires  that  the  decision  maker  evaluate  the  value  increments 
associated  with  “swinging”  from  the  least  preferred  scenario  to  the  most  preferred 
scenario  of  each  evaluation  measure  (Kirkwood,  1997).  The  value  increments  are  then 
scaled  as  multiples  of  the  smallest  value  increment.  The  second  technique  is  very  similar 
but  begins  with  the  decision  maker  making  a  subjective  detennination  as  to  which 
measure  within  that  tier  of  the  branch  is  least  important.  Once  that  is  detennined,  the 
decision  maker  subjectively  determines  the  importance  of  the  other  measures  in  multiples 
of  the  least  important  one.  With  both  techniques,  the  importance  ratings  are  combined 
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algebraically  in  a  summation  equation  equal  to  one  (since  the  weights  within  a  tier  of  the 
same  branch  must  equal  one).  The  equation  is  then  solved  to  detennine  the  weight  for 
the  evaluation  measure  with  the  smallest  value  increment  (or  least  importance).  The 
weights  for  the  remaining  values  (or  measures)  are  detennined  from  the  previous 
mathematical  relationship  (i.e.,  the  summation  equation). 

2.9.3.5.2  Global  Weighting 

Global  weighting,  often  called  overall  weighting,  refers  to  the  relative  importance 
assigned  to  values  or  measures  across  an  entire  tier;  it  shows  how  each  individual  value 
or  measure  contributes  to  the  overall  objective  (Katzer,  2002).  Although  the  same 
techniques  that  are  used  for  local  weighting  can  be  applied  to  global  weighting,  the  most 
common  practice  is  to  assign  local  weights  and  then  work  mathematically  through  the 
hierarchy  to  determine  the  related  global  weights.  Global  weights  are  often  used  as  a 
“double  check”  to  ensure  the  PDM  understands  how  the  local  weights  are  affecting  the 
overall  model.  To  calculate  global  weights  in  a  top-down  manner,  simply  multiply  the 
local  weights  of  each  value  or  measure  by  the  local  weight  of  every  value  above  it.  For 
example,  to  find  the  global  weight  of  power  in  the  truck  buying  example,  its  local  weight 
of  0.400  would  be  multiplied  by  the  local  weight  of  performance  (0.300)  and  the  local 
weight  of  the  fundamental  objective  ( 1 .000).  This  would  result  in  a  global  weight  of 
0.120.  Since  the  local  weight  of  the  horsepower  measure  is  1.000,  its  global  weight  is 
also  0.120.  This  is  illustrated  in  Figure  8. 
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Figure  8.  Example  Hierarchy  with  Global  Weights 
(Global  Weights  in  Parentheses) 


2.9. 3.6  Step  6  -Alternative  Generation 

Once  the  hierarchy  has  been  fully  developed,  the  next  step  is  to  generate 
alternatives.  The  process  of  creating  the  hierarchical  model  can  be  very  beneficial  to  the 
decision  maker  at  this  point;  the  accomplishment  of  the  previous  steps  helps  foster 
understanding  of  the  problem,  insight,  and  creativity,  all  of  which  can  aid  in  development 
of  new  and  better  alternatives.  Previous  experiences  can  also  be  used  to  develop 
alternatives  in  a  process  called  associative  reasoning,  which  is  simply  associating  the 
current  situation  with  relevant  past  ones  and  using  those  past  experiences  to  develop 
thoughts  and  ideas  (Kirkwood,  1997).  If  alternatives  are  difficult  to  develop,  a  good  tool 
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to  use  is  a  strategy  table  (Kirkwood,  1997;  Weir,  2003).  These  are  tables  that  outline 
pieces  of  alternatives,  measures,  and  other  parts  of  the  puzzle  that  allow  a  user  to 
visualize  new  strategies  suitable  to  the  problem.  Another  technique  that  can  be  used  is  to 
identify  value  gaps  in  the  current  alternatives  (Weir,  2003).  If  the  alternatives,  or  even 
just  the  top  alternative,  could  improve  its  rating  by  improving  in  a  particular  area,  then 
there  is  a  value  gap  in  that  area.  A  new  alternative  or  modification  to  the  present 
alternative  can  then  be  explored  to  exploit  the  value  gap.  If  there  are  too  many 
alternatives  to  choose  from,  screening  criteria  can  be  used  to  reduce  the  number  of 
alternatives  considered  to  a  more  feasible  level.  Mathematical  optimization  can  also  be 
used  and  is  most  frequently  accomplished  with  linear  integer  programming  (Kirkwood, 
1997). 

2.9.3. 7  Step  7  -  Alternative  Scoring 

Once  the  alternatives  have  been  generated,  the  next  step  is  to  score  them. 
Although  this  step  appears  relatively  straightforward,  it  can  be  one  of  the  most  difficult  to 
accomplish  if  the  model  has  been  poorly  developed.  This  is  where  creating  a  hierarchy 
that  fits  the  completeness,  nonredundancy,  independence,  operability,  and  “small”  size 
criteria  can  pay  big  dividends.  Collecting  the  data  to  score  the  alternatives  can  be  an 
immensely  difficult  and  time  consuming  process  if  the  value  hierarchy  has  numerous 
measures,  measures  with  difficult  to  obtain  data,  or  measures  with  ambiguous  scoring 
criteria.  Therefore,  following  a  few  simple  rules  can  help  smooth  this  step  and  ensure 
proper  scoring.  First,  always  properly  document  where  scoring  data  was  obtained  so  that 
the  scoring  can  be  repeated  or  tested.  Second,  the  data  should  also  be  scored  blindly 
without  knowing  how  each  score  affects  the  “value”  units  earned.  Finally,  it  is  extremely 


55 


helpful  to  score  one  measure  at  a  time  across  all  the  alternatives;  this  will  help  eliminate 
bias  by  maintaining  consistent  application  of  the  evaluation  criteria. 

2. 9. 3. 8  Step  8  -  Deterministic  Analysis 

Deterministic  analysis  requires  the  use  of  an  equation  that  combines  the  value 
“points”  for  each  alternative  (translated  from  the  scoring  data  by  the  SDVFs)  with  the 
weights  determined  by  the  decision  maker.  The  “value”  points  and  weights  are  paired  to 
give  an  aggregate  score  for  each  alternative  that  can  be  used  for  a  deterministic  (rack  and 
stack)  analysis  of  the  alternatives.  The  VFT  process  typically  uses  an  additive  value 
function  (summation)  to  detennine  the  aggregate  score.  The  required  conditions  for 
using  an  additive  value  function  are  a  valid  SVDF  associated  with  each  measure,  a 
weight  assigned  to  each  measure,  and  a  summation  of  weights  that  equals  one.  When 
these  conditions  are  met,  the  additive  value  function  can  be  represented  as: 

vO)  =  XAi'vi  (*.■) 

1=1 

where  v(x)  is  the  value  function  (i.e.,  aggregate  score),  vi(x)l  is  the  individual  measure 
value  converted  from  the  scoring  data  by  the  SDVF,  and  A;  is  the  global  weight  for  each 
respective  measure  (Kirkwood,  1997). 

2. 9. 3. 9  Step  9  -  Conduct  Sensitivity  Analysis 

When  the  initial  ranking  of  the  alternatives  is  completed  by  the  detenninistic 
analysis,  additional  insight  is  available  to  the  decision  maker  through  the  use  of 
sensitivity  analysis.  This  is  one  of  the  great  advantages  of  using  a  model;  it  allows  the 
analyst  to  detennine  how  changes  in  model  assumptions  impact  the  deterministic  analysis 
of  the  alternatives  (Kirkwood,  1997).  One  of  the  most  common  applications  of 


56 


sensitivity  analysis  is  to  look  at  how  changes  to  the  assigned  weights  affect  the  results. 
Since  the  weights  reflect  the  relative  importance  of  the  values  in  the  hierarchy,  it  is  often 
useful  to  show  the  decision  maker  how  the  use  of  different  weighting  schemes  might 
affect  the  ranking  of  the  alternatives.  In  addition  to  providing  for  a  more  informed 
decision,  this  might  also  indicate  the  need  for  extra  research  in  a  particular  area  or  the 
deletion  of  non-sensitive  values  from  the  hierarchy.  The  most  common  method  of 
sensitivity  analysis  is  varying  the  weight  of  one  value  while  holding  the  weights  of  all 
other  values  proportionally  constant  and  ensuring  the  weights  sum  to  one.  Sensitivity 
analysis  might  also  involve  changing  the  weights  on  one  tier  of  a  branch  as  a  group  while 
still  ensuring  the  weights  in  that  tier  sum  to  one.  More  involved  types  of  sensitivity 
analysis  also  exist  but  are  outside  the  scope  of  this  research. 

2.9.3.10  Step  10-  Presentation  of  Results 

After  the  analysis  has  been  completed,  the  results  are  presented  to  the  decision 
maker  by  the  analyst  in  a  clear,  understandable  manner.  The  analysis  should  be  tailored 
to  the  questions  initially  asked  (i.e.,  fundamental  objective)  and  the  insights  the  decision 
maker  is  attempting  to  gain.  At  this  stage,  it  is  imperative  for  the  analyst  and  decision 
maker  to  remember  that  the  VFT  process  is  designed  only  to  provide  insight  and  clarity 
about  the  decision;  it  is  not  designed  to  make  the  decision  itself.  The  final  decision 
should  always  be  at  the  discretion  of  the  decision  maker  regardless  of  the  results  of  the 
analysis. 
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Chapter  3.  Methodology 


The  United  States  Air  Force  has  been  directed  by  the  Secretary  of  Defense  to 
privatize  all  available  utility  systems.  To  accomplish  this  objective  in  the  most  efficient 
and  effective  manner,  it  is  necessary  to  develop  a  source  selection  process  that  is 
objective,  repeatable,  and  defendable.  The  value  focused  thinking  (VFT)  methodology 
provides  a  process  to  detennine  the  values  and  measures  of  the  Air  Force  regarding 
utilities  privatization  and  is  very  applicable  because  of  the  number  of  subjective  factors 
that  need  to  be  quantified  in  order  to  develop  an  objective  view  of  the  source  selection 
decision.  The  VFT  methodology  also  allows  weights  of  importance  to  be  assigned  to 
each  value  and  measure,  thereby  allowing  multiple  competing  values  to  be  traded  off 
against  each  other  during  the  decision  making  process.  This  chapter  examines  how  Steps 
1  through  7  of  the  VFT  process  were  applied  in  order  to  create  a  model  that  objectively 
quantifies  the  source  selection  decision. 

3,1  Step  1:  Problem  Identification 

The  first  step  in  the  10-Step  Value  Focused  Thinking  (VFT)  methodology  is  to 
identify  the  problem  in  terms  of  an  objective  question.  Discussions  with  the  Air  Force 
Civil  Engineer  Support  Agency  (AFCESA)  revealed  that  gaps  existed  in  the  utilities 
privatization  (UP)  arena  in  determining  a  “methodology  for  awarding,  implementing, 
measuring  and  verifying  UP  contractor  performance  in  performance-based  UP  contracts 
(Stahl,  2003).”  It  was  then  concluded  that  VFT  was  an  appropriate  methodology  to 
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develop  a  tool  for  analyzing  contractor  proposals  during  competitive  utilities  privatization 
solicitations. 

In  order  to  keep  the  scope  of  this  thesis  manageable,  it  was  decided  that  the  model 
would  be  designed  specifically  for  analyzing  the  source  selection  decision  for  privatizing 
an  electrical  system.  Therefore,  the  identified  problem  in  the  form  of  a  general  objective 
question  became:  “Do  we  award  the  electrical  system  at  Air  Force  Base  X  to  Contractor 
A,  B,  or  C?”  This  objective  question,  used  to  guide  the  process  of  building  the  model, 
incorporated  the  specific  contractors  being  evaluated  and  the  specific  base  where  the 
privatization  effort  was  underway. 

3.2  Step  2:  Create  the  Value  Hierarchy 

With  the  problem  clearly  identified,  the  next  step  was  to  solicit  the  values  relating 
to  the  fundamental  objective  and  logically  group  them  into  a  hierarchy.  Therefore  the 
analyst  created  an  affinity  diagram  of  values  obtained  from  literature  and  based  on 
experience  that  may  influence  a  utilities  privatization  source  selection  decision.  As 
shown  in  Figure  9,  four  top  values  emerged  from  the  affinity  grouping  process:  Cost, 
Responsiveness,  Reliability,  and  Quality.  Responsiveness  and  Reliability  were  then 
grouped  under  the  single  value  of  Mission  Capability,  leaving  the  hierarchy  with  three 
values  on  the  first  tier:  Cost,  Mission  Capability,  and  Quality. 
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Figure  9.  Initial  Tiers  of  Strawman  Hierarchy 


To  decompose  the  values,  the  most  recent  revision  to  the  Request  for  Proposal 
(RFP)  template  (February  of  2003)  was  analyzed.  Every  statement  that  reflected  a 
potential  desired  value  was  recorded;  the  resulting  values  that  were  integrated  into  the 
model  are  listed  in  Table  11.  This  list  confirmed  that  the  first-tier  values  developed  from 
the  affinity  diagram  (Cost,  Mission  Capability,  and  Quality)  were  still  applicable  and 
accurate  representatives  for  the  model.  Subsequently,  the  remainder  of  the  value 
hierarchy  was  generated  from  the  values  identified  during  the  affinity  grouping  process 
and  the  RFP  review.  The  resulting  “strawman”  hierarchy,  developed  using  the  gold 
standard,  is  shown  in  Figures  10,  11,  12,  and  13. 
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Table  11.  Potential  Values  Gleaned  from  RFP  Template 


•  Purchase  Price 

•  Recoverable  Portion  of  the  Purchase  Price 

•  #  of  Months  Credit  will  be  Realized 

•  %  Points  Above  or  Below  U.S.  Treasury  Bond  Rate 

•  Transition  Period  Cost 

•  #  of  Transition  Days 

•  Fixed  Monthly  Charge 

•  Tariff  Rate 

•  Initial  Capital  Upgrades 

•  Capital  Investments  (Upgrades  and  Renewals) 

•  Licenses,  Pennits,  or  Certifications 

•  Compliance  with  Federal,  State,  Interstate,  and  Local  Laws 

•  Quality  Management  Plan 

•  Submetering  (Installation,  Maintenance,  Reading,  Calibration,  and  Reporting) 

•  Commodity  Management 

•  Energy  and  Water  Conservation 

•  Energy  and  Water  Saving  Projects  (Current  and  Future) 

•  Efficiency  Upgrades  Proposed 

•  Maintain  Record  Drawings  of  Facilities 

•  Employees  Not  a  Potential  Threat 

•  Employees  Understand,  Read,  Write,  and  Speak  English 

•  Employees  Meet  Applicable  Federal,  State,  Local,  and  Installation  Certification, 
Licensing,  and  Medical  Requirements 

•  Employees  Cannot  Generate  Conflicts  of  Interest 

•  Military  Employees  Cannot  Constitute  Excuse  for  Nonperformance 

•  National  Agency  Check  Infonnation 

•  Emergency  Restoration  Plan  Adhered  To 

•  Service  Interruption/Contingency  Plan 

•  Credit  to  AF  for  not  Meeting  Restoration  Times  (N/A  for  Regulated  Utility) 

•  Service  Request  System 

•  Emergency,  Urgent,  and  Routine  Service  Responses 

•  Work  Coordination  and  Notification 

•  Service  Record  Retainage 

•  Exercises/Contingencies 

•  Excavation  Permit  Process 

•  Attendance  of  Contracting  Meetings 

•  Environmental  Pennits  and  Compliance 

•  Spill  Contingency  Plan 

•  HazMat  and  HazWaste  Minimization  Plan 
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Figure  10.  Cost  Branch  of  Strawman  Hierarchy 
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Figure  1 1 .  Reliability  Branch  of  Strawman  Hierarchy 
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Figure  12.  Responsiveness  Branch  of  Strawman  Hierarchy 
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Figure  13.  Quality  Branch  of  Strawman  Hierarchy 
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The  “strawman”  hierarchy  was  used  to  facilitate  discussions  with  the  PDM,  who 
was  an  AFCESA  representative  experienced  in  evaluating  source  selection  proposals. 
Over  a  span  of  several  months,  the  analyst  worked  with  the  PDM  to  develop  a  new 
hierarchy  that  the  PDM  thought  better  reflected  the  Air  Force’s  values  regarding  utilities 
privatization  source  selection.  The  top  three  values  (i.e.,  first-tier)  in  the  hierarchy,  which 
represent  the  most  important  aspects  of  the  fundamental  objective,  were  identified  by  the 
PDM  as:  Capability/Risk,  Cost,  and  Past  Perfonnance.  The  final  hierarchy,  developed 
using  the  platinum  standard  and  consisting  of  16  values  and  26  measures,  is  shown  in 
Figure  14. 
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The  Capability/Risk  value  is  concerned  with  the  overall  capability  of  and  the 
feasibility  of  the  plans,  work,  and  costs  outlined  in  the  proposal.  This  is  extremely 
important  to  the  Air  Force  since  one  of  the  core  reasons  behind  privatization  is  to 
increase  the  quality  and  efficiency  of  the  service  being  provided.  The  Cost  value  is 
focused  on  the  net  present  value  (NPV)  of  the  expected  stream  of  cash  flows  over  the 
lifetime  of  the  contract.  Once  again,  cost  savings  is  associated  with  the  utilization  of  a 
private  firm’s  greater  expertise  and  efficiency  and  is  one  of  the  top  reasons  given  for 
privatizing  a  system.  Past  Performance  is  also  a  highly  desirable  value  since  it  is  an 
indicator  of  the  Air  Force’s  confidence  in  the  contractor’s  ability  to  perform  as  promised. 
This  confidence  level  is  very  important  due  to  the  length  of  most  utilities  privatization 
contracts.  The  lower  tier  values  identified  for  each  of  these  top  tier  values  adds 
specificity  in  terms  of  what  the  Air  Force  deems  important  in  evaluating  contractor 
proposals  in  a  utilities  privatization  source  selection  scenario.  Explanations  of  all  values 
within  the  hierarchy  are  presented  in  Appendix  A. 

3.3  Step  3:  Develop  Evaluation  Measures 

After  the  value  hierarchy  was  created,  the  next  step  was  to  create  and  define 
evaluation  measures  that  allow  for  the  measurement  of  the  attainment  of  the  objectives 
and  values.  The  measures  provided  the  ability  to  quantify  various  aspects  of  the 
hierarchy  and  allowed  for  a  more  objective  decision  making  process.  To  make  the 
hierarchy  operable  and  complete  while  preserving  independence  and  nonredundancy 
among  the  evaluation  considerations,  the  evaluation  measures  were  also  generated  using 
the  RFP  template  and  PDM  inputs  to  determine  what  data  could  be  gleaned  from 
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proposals  if  submitted  correctly  and  completely.  At  the  same  time,  every  effort  was 
made  to  keep  the  hierarchy  as  small  as  possible  to  maintain  its  ease  of  use. 

In  creating  the  measures  used  in  this  model,  the  intention  was  to  have  as  many 
measures  with  natural,  direct  scales  as  possible  and  as  few  measures  with  constructed, 
proxy  scales  as  possible.  This  was  a  very  difficult  proposition,  however,  since  proposals 
typically  do  not  have  historical  or  hard  data  for  most  of  the  measures.  This  meant  that  a 
great  deal  of  categorical  measures  with  constructed,  proxy  scales  had  to  be  used  (3 
Natural,  Direct;  4  Natural,  Proxy;  19  Constructed,  Proxy).  Table  12  summarizes  the 
evaluation  measures  that  were  developed  in  the  model.  To  develop  the  scales  for  each 
measure,  the  PDM  decided  whether  categorical  or  quantitative  scales  were  appropriate 
and  defined  the  upper  and  lower  bounds  for  each  of  the  measures.  The  definitions  for 
each  measure  are  listed  in  Appendix  B.  As  an  example,  the  definitions  used  for  “Interest 
Coverage”  and  “Compliance  with  Federal,  Interstate,  State,  and  Local  Laws”  are  shown 
in  Tables  13  and  14  respectively. 
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Table  12.  Evaluation  Measures  Summary 


Third  Tier  Value 

Measure 

Lower 

Bound 

Upper  Bound 

Capability/ 

Feasibility 

Economic 

Feasibility 

N/A 

Interest  Coverage 

Unqualified 

Unconditionally 

Qualified 

FFO  to  Interest  Ratio 

Unqualified 

Unconditionally 

Qualified 

FFO  to  Total  Debt  Percentage 

Unqualified 

Unconditionally 

Qualified 

Total  Debt  to  Total  Capital  Ratio 

Unqualified 

Unconditionally 

Qualified 

Disaster  Recovery  Ratio 

Unqualified 

Unconditionally 

Qualified 

Quality 

Compliance  with 
Laws  and 
Requirements 

Compliance  w/Fed.,  State, 

Interstate,  &  Local  Laws  Adequately 

None 

Exceptional 

Compliance  with  Base 

Requirements  Adequately  Adressed 

None 

Exceptional 

Licensing,  Permitting,  & 

Certification  Adequately  Addressed 

None 

Exceptional 

Effective  Base 
Interaction 

Line  Locating;  Updating  Maps  & 
Drawings  Adequately  Addressed 

None 

Exceptional 

Effective 

Management 

Best  Engr  &  Mgmt  Practices 
Adequately  Addressed 

None 

Exceptional 

Level  of  Awards  and  Certificates 

None 

National 

Safety 

Certification  of  Employees 
Adequately  Addressed 

No 

Yes 

English  Skills  of  Employees 
Adequately  Addressed 

No 

Yes 

Security 

NAC  Information 

Gathering/Providing  Adequately 

No 

Yes 

Reliability 

N/A 

Capital  Upgrades  &  Renewals  and 
Replacements  Plan  Adequacy 

None 

Exceptional 

Service/ 
Interruption  & 
Contingency 
Responsiveness 

N/A 

Flazmat/Flazwaste  Minimization 
Adequately  Addressed 

None 

Exceptional 

Service  Providance  Adequately 

None 

Exceptional 

Spill  Contingency  Plan  Adequacy 

None 

Exceptional 

Smooth 

Transition 

N/A 

Days  to  Transition 

90 

365 

Transition  Plan  Adequacy 

None 

Exceptional 

Cost 

N/A 

N/A 

CEA  NPV  %  of  Should  Cost 

100 

75 

Past 

Perfomance 

Competency 

N/A 

%  of  Positive  Past  Performance 

66 

100 

Recency 

N/A 

#  of  Similar  Scope  Systems  in  Last 

0 

5 

#  of  Similar  Scope  Systems  in  Last 

0 

5 

Relevancy 

N/A 

%  of  Similar  Scope  Systems  in  Last 

0 

50 

%  of  Similar  Scope  Systems  in  Last 

0 

50 
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Table  13.  Category  Definitions  for  Interest  Coverage  Measure 


Category 

Category  Definition 

Unqualified 

1 .0  x  or  less  for  Investor  Owned  system.  1 .0  x  for  less  for 
Municipal  or  Cooperative  Owned  system. 

Conditionally 

Qualified 

1.0-2. 5  x  for  Investor  Owned  system.  1.0- 1.5  x  for  Municipal 
or  Cooperative  Owned  system. 

Unconditionally 

Qualified 

2.5  x  or  greater  for  Investor  Owned  system.  1 .5  x  or  greater 
for  Municipal  or  Cooperative  Owned  system. 

Table  14.  Category  Definitions  for  Compliance  with  Federal,  Interstate,  State,  and  Local 

Laws  Measure 


Category 

Category  Definition 

None 

Compliance  with  these  laws  is  not  addressed  in  the  proposal. 

Minimal 

Proposal  shall  address  all  applicable  federal,  state,  interstate, 
and  local  laws/regulations  that  must  be  complied  with  in  order 
to  provide  this  service  including  those  requirements  relating  to 
health,  safety,  and  the  environment. 

Good 

Proposal  expands  to  address  specifically  how  each  applicable 
law  will  affect  each  portion  of  the  operation  and  how  each 
facet  will  be  managed  to  avoid  non-compliance. 

Exceptional 

Proposal  includes  all  previous  and  also  addresses  plan  for 
modifying  service  practices  as  applicable  laws  are  dropped, 
added,  or  amended.  Should  include  discussion  of  pending 
laws  and  changes  to  laws. 
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3.4  Step  4:  Create  Value  Functions 

Once  the  measures  for  quantifying  the  lower  tier  values  were  determined,  the  next 
step  was  to  define  the  single  dimension  value  functions  (SDVFs)  for  each  of  the 
measures.  Since  the  x-axis  bounds  were  identified  in  Step  3,  each  SDVF  transforms  the 
x-axis  scores  into  y-axis  “value”  units.  This  quantifies  the  subjective  pieces  of  the  model, 
thereby  allowing  objective  analysis  to  take  place  later  in  the  process.  As  defined  in 
Chapter  2,  the  value  functions  have  “value”  units  that  range  from  0  to  1  with  the  least 
preferred  category  or  score  for  that  measure  receiving  a  0  and  the  most  preferred  category 
or  score  for  that  measure  receiving  a  1.  The  SDVFs  were  determined  iteratively  in  two 
ways.  Continuous  functions  were  defined  by  using  Excel  macros  to  create  visual 
functions  within  spreadsheets  that  could  be  adjusted  by  the  PDM  for  inclusion  in  the 
model.  Discrete  functions  were  defined  directly  by  the  PDM.  Examples  of  each  are 
provided  in  the  following  figures. 

Figure  15  shows  the  exponential,  monotonically  decreasing  SDVF  for  the  “CEA 
NPV  %  of  Should  Cost”  measure  and  Figure  16  shows  the  discrete,  monotonically 
increasing  SDVF  for  the  “Interest  Coverage”  measure.  The  SDVFs  for  all  of  the 
measures  are  included  in  Appendix  B. 


72 


Figure  15.  SDVF  for  CEA  NPV  %  of  Should  Cost 


Label 

Value 

Unqualified 

Qualified 

Unconditionally  Qualified 

0.000 

0.600 

1.000 

1 

Figure  16.  SDVF  for  Interest  Coverage 
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3.5  Step  5:  Weight  the  Value  Hierarchy 

Once  the  measures  and  SDVFs  were  determined,  each  of  the  values  and  measures 
were  assigned  a  relative  importance  (or  weight),  thereby  allowing  the  PDM  to 
differentiate  between  the  significance  of  each  of  the  values  represented  in  the  hierarchy. 
At  the  request  of  the  PDM,  the  local  weights  were  initially  solicited  from  a  subject  matter 
expert  (SME)  and  then  forwarded  to  the  PDM  for  refinement.  The  SME  weighted  each 
component  using  a  top-down  approach  and  was  instructed  to  do  this  by  using  the  100 
marbles  technique  to  facilitate  the  process.  The  weights  returned  by  the  PDM  were  then 
used  to  determine  the  global  weights.  The  PDM  reviewed  the  global  weights  to  ensure  a 
complete  understanding  of  their  global  impact. 

Figure  17  shows  the  complete  hierarchy  with  the  local  and  global  weights  (global 
weights  in  parentheses)  for  each  value.  Table  15  provides  the  local  and  global  weights 
for  each  of  the  measures;  Table  16  shows  a  cumulative  look  at  the  global  weights  by 
ranking  the  measures  from  highest  global  weight  to  lowest  global  weight.  The  three 
measures  with  the  highest  global  weights  are  the  “CEA  NPV  %  of  Should  Cost”  measure 
(0.250),  the  “Capital  Upgrades  and  Renewals  &  Replacements  Plan  Adequacy”  measure 
(0. 125),  and  the  “%  of  Positive  Past  Performance  Reviews”  measure  (0. 100).  This  makes 
intuitive  sense  because  decreased  cost  and  increased  reliability  are  two  of  the  major 
benefits  of  privatization,  while  past  perfonnance  reviews  are  the  most  powerful  indicator 
of  future  performance.  These  three  measures  comprise  47.5%  of  the  total  value 
represented  in  the  model.  Table  16  also  shows  that  the  top  17  measures  hold 
approximately  90%  of  the  global  weight  in  the  model. 


74 


-J 

L/l 


Figure  17.  Value  Hierarchy 


Local  and  Global  Weights  (Global  in  Parentheses) 


Table  15.  Local  and  Global  Weights  of  Evaluation  Measures 


First  Tier  Value 

■|iTT7S||g« 

Measure 

Weights 

Capability/ 

Feasibility 

Economic 

Feasibility 

N/A 

Interest  Coverage 

0.200 

0.020 

FFO  to  Interest  Ratio 

0.200 

0.020 

FFO  to  Total  Debt  Percentage 

0.200 

0.020 

Total  Debt  to  Total  Capital  Ratio 

0.200 

0.020 

Disaster  Recovery  Ratio 

0.200 

0.020 

Quality 

Compliance 
with  Laws  and 
Requirements 

Compliance  w/Fed.,  State,  Interstate,  &  Local  Laws 
Adequately  Addressed 

0.350 

0.010 

Compliance  with  Base  Requirements  Adequately 
Adressed 

0.300 

0.009 

Licensing,  Permitting,  &  Certification  Adequately 
Addressed 

0.350 

0.010 

Effective 

Base 

Line  Locating;  Updating  Maps  &  Drawings  Adequately 
Addressed 

1.000 

0.010 

Effective 

Management 

Best  Engr  &  Mgmt  Practices  Adequately  Addressed 

0.600 

0.009 

Level  of  Awards  and  Certificates 

0.400 

0.006 

Safety 

Certification  of  Employees  Adequately  Addressed 

0.600 

0.014 

English  Skills  of  Employees  Adequately  Addressed 

0.400 

0.009 

Security 

NAC  Information  Gathering/Providing  Adequately 
Addressed 

1.000 

0.023 

Reliability 

N/A 

Capital  Upgrades  &  Renewals  and  Replacements 

Plan  Adequacy 

1.000 

0.125 

Service/ 
Interruption  & 
Contingency 
Responsiveness 

N/A 

Hazmat/Hazwaste  Minimization  Adequately 

Addressed 

0.300 

0.038 

Service  Providance  Adequately  Addressed 

0.350 

0.044 

Spill  Contingency  Plan  Adequacy 

0.350 

0.044 

Smooth 

Transition 

N/A 

Days  to  Transition 

0.400 

0.020 

Transition  Plan  Adequacy 

0.600 

0.030 

Cost 

N/A 

N/A 

CEA  NPV  %  of  Should  Cost 

1.000 

0.250 

Past 

Perfomance 

Competency 

N/A 

%  of  Positive  Past  Performance  Reviews 

1.000 

0.100 

Recency 

N/A 

#  of  Similar  Scope  Systems  in  Last  1-3  Years 

0.600 

0.045 

#  of  Similar  Scope  Systems  in  Last  4-5  Years 

0.400 

0.030 

Relevancy 

N/A 

%  of  Similar  Scope  Systems  in  Last  1-3  Years 

0.600 

0.045 

%  of  Similar  Scope  Systems  in  Last  4-5  Years 

0.400 

0.030 
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Table  16.  Cumulative  Chart  of  Measure  Global  Weights 


Measure 

Local 

Weights 

Global 

Weights 

cumulative 

Global 

Weight 

CEA  NPV  %  of  Should  Cost 

1.000 

0.250 

0.250 

Capital  Upgrades  &  Renewals  and  Replacements 

Plan  Adequacy 

1.000 

0.125 

0.375 

%  of  Positive  Past  Performance  Reviews 

1.000 

0.100 

0.475 

#  of  Similar  Scope  Systems  in  Last  1-3  Years 

0.600 

0.045 

0.520 

%  of  Similar  Scope  Systems  in  Last  1-3  Years 

0.600 

0.045 

0.565 

Service  Providance  Adequately  Addressed 

0.350 

0.044 

0.609  \ 

Spill  Contingency  Plan  Adequacy 

0.350 

0.044 

0.653 

Hazmat/Hazwaste  Minimization  Adequately 

0.300 

0.038 

0.691 

Transition  Plan  Adequacy 

0.600 

0.030 

0.721 

#  of  Similar  Scope  Systems  in  Last  4-5  Years 

0.400 

0.030 

0.751 

%  of  Similar  Scope  Systems  in  Last  4-5  Years 

0.400 

0.030 

0.781 

NAC  Information  Gathering/Providing  Adequately 
Addressed 

1.000 

0.023 

0.804 

Interest  Coverage 

0.200 

0.020 

0.824 

FFO  to  Interest  Ratio 

0.200 

0.020 

0.844 

FFO  to  Total  Debt  Percentage 

0.200 

0.020 

0.864 

Total  Debt  to  Total  Capital  Ratio 

0.200 

0.020 

0.884 

Disaster  Recovery  Ratio 

0.200 

0.020 

0.904 

Days  to  Transition 

0.400 

0.020 

0.924 

Certification  of  Employees  Adequately  Addressed 

0.600 

0.014 

0.938 

Compliance  w/Fed.,  State,  Interstate,  &  Local  Laws 
Adequately  Addressed 

0.350 

0.010 

0.948 

Licensing,  Permitting,  &  Certification  Adequately 
Addressed 

0.350 

0.010 

0.958 

Line  Locating;  Updating  Maps  &  Drawings  Adequately 
Addressed 

1.000 

0.010 

0.968 

Compliance  with  Base  Requirements  Adequately 

0.300 

0.009 

0.977 

Best  Engr  &  Mgmt  Practices  Adequately  Addressed 

0.600 

0.009 

0.986 

English  Skills  of  Employees  Adequately  Addressed 

0.400 

0.009 

0.995 

Level  of  Awards  and  Certificates 

0.400 

0.006 

1.001 
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3.6  Step  6:  Alternative  Generation 

Once  the  value  hierarchy  and  associated  measures  have  been  weighted,  the 
emphasis  shifts  to  identifying  alternatives  to  evaluate  against  the  hierarchy.  Evaluation 
and  analysis  of  these  alternatives  will  give  the  decision  maker  additional  insight  into  the 
decision  problem.  For  this  research  effort,  the  original  agenda  was  to  obtain  proposals 
from  two  Air  Force  bases  that  were  either  undergoing  or  had  completed  a  privatization 
effort  for  their  electrical  system.  However,  it  was  discovered  that  no  active,  reserve,  or 
air  national  guard  base  had  privatized  their  electrical  system  utilizing  the  current  RFP 
solicitation  process  at  the  time  of  this  research.  Additionally,  there  were  no  bases 
currently  privatizing  their  electrical  systems.  Therefore,  the  model  was  tested  against 
notional  data  sets  generated  using  a  random  number  generator. 

Assuming  a  uniform  distribution,  a  random  number  generator  was  used  to 
create  500  data  sets,  each  consisting  of  26  data  points  (26  measures).  To  reduce  this 
number  to  a  more  manageable  level,  filters  were  used  to  maximize  the  value  of  each  of 
the  first  and  second-tier  values  (a  total  of  1 1  values).  Maximizing  two  of  the  values,  Cost 
and  Recency,  produced  the  same  results;  thus,  there  were  10  proposals  (instead  of  1 1) 
identified  for  detailed  analysis  using  this  filtering  process. 

As  a  general  rule,  the  analyst  maximized  the  measure(s)  beneath  the  value  being 
maximized  in  rank  order  according  to  their  local  weights  and  then  continued  the  process 
using  the  global  weights  of  all  other  measures.  For  example,  to  maximize  the  Reliability 
value,  only  proposals  that  maximized  the  score  on  the  “Capital  Upgrades  and  Renewals 
and  Replacements  Plan  Adequacy”  measure  were  included.  This  reduced  the  number  of 
proposals  from  500  to  134.  Since  there  were  no  other  measures  under  the  Reliability 
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value,  the  analyst  used  the  rank  order  list  of  global  weights  to  continue  the  filtering 
process.  The  measure  with  the  highest  global  weight  was  “CEA  NPV  %  of  Should 
Cost.”  Maximizing  this  measure  reduced  the  list  from  134  to  4  proposals.  The  measure 
with  the  next  highest  global  weighting  was  “%  of  Positive  Past  Performance  Surveys.” 
Maximizing  this  measure  reduced  the  list  to  a  single  proposal.  This  general  process  was 
used  for  each  of  the  1 1  values  in  either  the  first  or  second  tier  of  the  hierarchy.  To 
facilitate  the  process,  the  measures  were  numbered  as  shown  in  Table  17.  The  results  of 
the  overall  filtering  process,  shown  in  Table  18,  represent  the  alternatives  to  be  scored  in 
the  next  step  of  the  VFT  process. 
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Table  17.  Numbering  System  for  Evaluation  Measures 


Measure 

Interest  Coverage 

Ml 

FFO  to  Interest  Ratio 

M2 

FFO  to  Total  Debt  Percentage 

M3 

Total  Debt  to  Total  Capital  Ratio 

M4 

Disaster  Recovery  Ratio 

M5 

Compliance  w/Fed.,  State,  Interstate,  &  Local  Laws  Adequately 

M6 

Compliance  with  Base  Requirements  Adequately  Adressed 

M7 

Licensing,  Permitting,  &  Certification  Adequately  Addressed 

M8 

Line  Locating;  Updating  Maps  &  Drawings  Adequately  Addressed 

M9  i 

Best  Engr  &  Mgmt  Practices  Adequately  Addressed 

M10 

Level  of  Awards  and  Certificates 

Mil 

Certification  of  Employees  Adequately  Addressed 

M12 

English  Skills  of  Employees  Adequately  Addressed 

M13 

NAC  Information  Gathering/Providing  Adequately  Addressed 

M14 

Capital  Upgrades  &  Renewals  and  Replacements  Plan  Adequacy 

M15 

Hazmat/Hazwaste  Minimization  Adequately  Addressed 

M16 

Service  Providance  Adequately  Addressed 

M17 

Spill  Contingency  Plan  Adequacy 

M18 

Days  to  Transition 

M19  j 

Transition  Plan  Adequacy 

M20 

CEA  NPV  %  of  Should  Cost 

M21 

%  of  Positive  Past  Performance  Reviews 

M22  I 

#  of  Similar  Scope  Systems  in  Last  1-3  Years 

M23 

#  of  Similar  Scope  Systems  in  Last  4-5  Years 

M24 

%  of  Similar  Scope  Systems  in  Last  1-3  Years 

M25 

%  of  Similar  Scope  Systems  in  Last  4-5  Years 

M26 
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Table  18.  Alternatives  Chosen  from  Notional  Data  Sets 


Alternative 

Value  Being 
Maximized 

Proposal 

Measures  Filtered 
(In  Order) 

A1 

Capability/Risk 

P89 

15,17,18,16,20,14,3 

A2 

Cost  (Recency) 

P389 

21,15,23&25 

(23,24,21) 

A3 

Past  Performance 

P488 

22,23,25 

A4 

Economic 

Feasibility 

P260 

1,2, 3, 4, 5 

A5 

Quality 

P354 

14,12,6,8,9,7 

A6 

Reliability 

P285 

15,21,22 

A7 

Service/Interruption 
&  Contingency 
Responsiveness 

P83 

17,18,16,21 

A8 

Smooth  Transition 

P297 

20,19 

A9 

Competency 

P216 

22,21 

A10 

Relevancy 

P333 

25,26 

3.7  Step  7:  Alternative  Scoring 

The  final  step  in  the  process  prior  to  performing  analysis  on  the  alternatives  is 
actually  scoring  the  alternatives.  For  this  research,  alternative  scoring  was  part  of  the 
alternative  generation  since  the  scores  were  used  to  generate  notional  data  sets.  A  full  list 
of  the  scores  for  each  randomly  generated  and  systematically  selected  proposal  is 
included  at  Appendix  C. 
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Chapter  4.  Results  and  Analysis 


This  chapter  presents  Steps  8  and  9  of  the  value  focused  thinking  (VFT)  process. 
In  Step  8,  a  rank-ordered  list  of  the  alternatives  is  generated  using  deterministic  analysis. 
Insight  is  also  obtained  regarding  the  values  having  the  greatest  impact  on  each 
alternative’s  final  score.  In  Step  9,  sensitivity  analysis  is  perfonned  by  varying  the 
weights  assigned  to  the  values  or  measures  and  evaluating  the  respective  changes  in  the 
deterministic  results.  The  sensitivity  analysis  gives  the  decision  maker  insight  into  how 
changes  in  the  model  weighting  might  affect  the  rank-ordering  of  the  alternatives. 
Sensitivity  analysis  also  gives  the  decision  maker  insight  into  which  values  and  measures 
cause  the  decision  to  be  particularly  sensitive  to  weight  changes;  it  also  depicts  how 
sensitive  the  decision  problem  is  to  those  weight  changes.  Since  notional  data  was  used 
for  this  research,  the  analysis  presented  in  this  chapter  is  provided  only  as  an  example  of 
the  potential  insight  to  be  gained  from  the  model  if  applied  to  a  real-world  source 
selection  scenario. 

4.1  Deterministic  Analysis 

As  previously  discussed,  the  deterministic  analysis  is  performed  using  an  additive 
value  function  which  mathematically  combines  the  “value”  points  earned  by  each 
measure  and  the  weights  specified  in  the  model  to  detennine  a  rank  order  of  the 
alternatives  based  on  a  total  value  scale  of  0  to  1.  Figures  18  and  19  show  the  overall 
ranking  of  the  alternatives  for  this  research,  with  the  “value”  points  being  earned  ranging 
from  0.806  for  Alternative  6  to  0.510  for  Alternative  4.  There  are  relatively  clear  breaks 
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between  most  of  the  alternatives  with  the  closest  point  of  contention  being  between 
Alternatives  3  and  8.  Alternative  6  would  appear  to  be  fairly  superior  based  on  this 
ranking  and  is  the  recommended  alternative  from  this  data  set. 

Figure  19  provides  additional  detail  regarding  how  each  of  the  alternatives  scored 
against  the  first-tier  values  of  Capability/Risk,  Cost,  and  Past  Performance.  Although 
Alternative  6  does  not  have  the  highest  score  against  any  of  the  three  first-tier  values,  it 
scores  well  against  all  of  them  and  the  balanced  results  give  it  a  higher  total  score  than 
any  of  the  other  alternatives.  Recall  that  the  Capability/Risk  value  has  a  global  weight  of 
0.500  and  accounts  for  half  of  the  potential  value  in  the  model.  Even  though  Alternative 
1  scored  the  highest  against  this  value,  it  is  ranked  second  overall.  Thus,  Figure  19  is  a 
good  illustration  of  the  tradeoffs  being  made  within  this  multi-objective  decision  analysis. 
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Figure  18.  Overall  Ranking  of  Alternatives 


Figure  19.  Overall  Ranking  with  First- tier  Breakout 
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For  another  look  at  the  overall  alternative  rankings,  Figure  20  provides  a  breakout 
of  the  top  12  measures  (which  account  for  80.4%  of  the  model’s  value)  with  the 
remaining  14  shown  as  a  lump  sum.  As  would  be  expected,  although  the  Cost  value  only 
has  a  global  weight  of  0.250,  the  only  measure  below  it,  “CEA  NPV  %  of  Should  Cost,” 
has  the  most  potential  influence  on  the  decision  because  its  global  weight  is  also  0.250, 
which  is  the  highest  global  weight  among  the  measures.  Another  interesting  point  is  that 
although  the  “CEA  NPV  %  of  Should  Cost”  and  “Capital  Upgrades  and  Renewals  and 
Replacements  Plan  Adequacy”  measures  account  for  a  great  deal  of  value  in  the  top  four 
alternatives,  their  combined  value  does  not  override  the  rest  of  the  measures.  This  again 
highlights  the  tradeoffs  inherent  in  this  type  of  decision  analysis  and  the  necessity  of 
creating  a  model  that  captures  all  of  the  relevant  values. 
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Alternative  #6 
Alternative  #1 
Alternative  #9 
Alternative  #2 
Alternative  #7 
Alternative  #5 
Alternative  #3 
Alternative  #8 
Alternative  #10 
Alternative  #4 


0.806 

0.762 

0.721 

0.687 

0.645 

0.603 

0.563 

0.557 

0.525 

0.510 


21.  CEANPV  %  of  Should  Cost 

15.  Captial  Upgrades  and  Renewals  &  Replacements  Plan  Adequacy 

22.  %  ofPositive  Past  Performance  Surveys 

25.  %  of  Similar  Scope  Systems  in  Last  1-3  Years 

23.  #  of  Similar  Scope  Systems  in  Last  1-3  Years 
17.  Service  Prov id ance  Adequately  Addressed 

1  8.  Spill  Contingency  Plan  Adequacy 

16.  Hazmat/Hazwaste  Minimization  Adequately  Addressed 

24.  #  of  Similar  Scope  Systems  in  Last  4-5  Years 

26.  %  of  Similar  Scope  Systems  in  Last  4-5  Years 
20.  Transition  Plan  Adequacy 

14.  NAC  Information  Gathering/Providing  Adequately  Addressed 
Other 


Figure  20.  Overall  Ranking  of  Alternatives  with  Measures  Breakout 
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In  addition  to  the  overall  ranking,  additional  insight  can  be  gained  from 
examining  the  detenninistic  analysis  results  for  each  of  the  values  in  the  hierarchy.  This 
information  could  be  used  when  initiating  discussions  and  seeking  clarifications  with 
contractors  during  source  selection  or  when  explaining  to  contractors  why  they  were  not 
awarded  the  utilities  privatization  contract.  For  example,  Figure  21  shows  the  scores  of 
the  alternatives  against  the  first-tier  value,  Capability/Risk,  broken  down  into  the  second- 
tier  values  beneath  it.  Based  on  this  value  alone,  Alternative  1  is  the  highest  ranked 
proposal.  Shown  in  this  manner,  further  insight  is  provided  regarding  the  lower  ranked 
alternatives.  For  instance,  suppose  the  contractor  for  Alternative  9  is  interested  in  its 
overall  third  place  ranking.  From  Figure  21,  it  can  be  seen  that  Alternative  9  scored 
poorly  against  the  Capability/Risk  value.  Figure  21  also  provides  further  insight  and 
shows  that  Alternative  9  scored  poorly  in  all  five  of  the  second-tier  values.  To  examine 
more  specifically  the  cause  for  these  poor  scores,  the  alternative  rankings  could  be  further 
refined  by  examining  the  impact  of  each  specific  measure.  From  Figure  22,  it  can  be 
seen  that  Alternative  received  zero  value  for  four  of  the  nine  measures  under  Quality. 
When  compared  with  the  top  ranked  alternatives  against  Capability/Risk  (Alternatives  1 
and  6),  the  “NAC  Information  and  Gathering  Adequately  Addressed”  measure  is 
highlighted  as  a  particular  weakness.  This  type  of  insight  would  prompt  reexamination  of 
the  scores  for  these  measures  as  well  as  provide  areas  for  discussion  and  clarification 
with  the  contractor. 
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Alternative 
Alternative  #1 
Alternative  #6 
Alternative  #10 
Alternative  #5 
Alternative  #4 
Alternative  #2 
Alternative  #8 
Alternative  #7 
Alternative  #9 
Alternative  #3 


■  Service/Interruption  &  Contingency  Responsiveness 

□  Reliability 

□  Quality 

□  Economic  Feasibility 
B!  Smooth  Transition 


Figure  21.  Alternative  Rankings  against  Capability/Risk  Value 
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Alternative 

Value 

Alternative 

#6 

0.890 

Alternative 

#5 

0.880 

Alternative 

#1 

0.846 

Alternative 

#4 

0.599 

Alternative 

#3 

0.540 

Alternative 

#7 

0.520 

Alternative 

#9 

0.465 

Alternative 

#2 

0.444 

Alternative 

#10 

0.365 

Alternative 

#8 

0.330 

■  14.  NAC  Information  Gathering/Providing  Adequately  Addressed 
12.  Certification  of  Employees  Adequately  Addressed 

□  8.  Licensing,  Pennitting,  &  Certification  Adequately  Addressed 

□  6.  Compliance  w/Fed,  State,  Interstate,  &  Local  Laws  Adequately  Addressed 
G3  9.  Line  Locating;  Updating  Maps  &  Drawings  Adequately  Addressed 

■  7.  Compliance  with  Base  Requirements  Adequately  Addressed 

■  13.  English  Skills  of  Employees  Adequately  Addressed 

■  10.  Best  Engr  &  Mgmt  Practices  Adequately  Addressed 

■  11.  Level  of  Awards  and  Certifications 


Figure  22.  Alternative  Rankings  against  Quality  Value  with  Measures  Breakout 


During  the  source  selection  process,  the  Source  Selection  Authority  (SSA)  often 
has  to  narrow  their  options  to  a  group  of  proposals  that  are  considered  the  most 
competitive.  This  group  of  proposals  represents  what  is  termed  the  competitive  range. 
Deterministic  analysis  could  be  used  to  establish  a  competitive  range  during  the  source 
selection  process.  If  the  competitive  range  was  based  on  the  costs  proposed  by  each 
contractor,  a  simple  deterministic  analysis  as  shown  in  Figure  23  may  indicate  that  a 
defendable  competitive  range  would  be  Alternatives  2,  6,  7,  9,  5.  There  is  a  distinct 
separation  between  these  five  alternatives  and  the  other  five.  However,  if  the  competitive 
range  was  based  on  the  Past  Performance  value,  the  task  would  be  slightly  more  difficult. 
As  illustrated  in  Figure  24,  the  top  two  Alternatives  (3  and  9)  score  much  higher  than  the 
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rest  of  the  alternatives;  furthermore,  seven  of  the  eight  remaining  alternatives  are  grouped 
together  without  much  separation  between  them.  Therefore,  the  use  of  Past  Perfonnance 
criteria  alone  would  result  in  either  a  very  narrow  or  very  wide  competitive  range  for  this 
set  of  alternatives. 


Figure  23.  Alternative  Rankings  against  Cost  Value 
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Alternative 

Value 

Alternative  #3 

0.930 

Alternative  #9 

0.854 

Alternative  #1 

0.500 

Alternative  #10 

0.496 

Alternative  #2 

0.484 

Alternative  #6 

0.455 

Alternative  #7 

0.431 

Alternative  #8 

0.414 

Alternative  #4 

0.413 

Alternative  #5 

0.174 

■  1 .  Competency 

□  2. 

2.  Recency 


3.  Relevancy 


Figure  24.  Alternative  Rankings  against  Past  Performance  Value 


4.2  Sensitivity  Analysis 

The  use  of  sensitivity  analysis  on  the  results  can  provide  additional  insight  into 
the  decision  problem.  Sensitivity  analysis  is  one  of  the  great  advantages  of  using  a 
model;  it  allows  the  analyst  to  determine  how  changes  in  model  assumptions  impact  the 
deterministic  analysis  of  the  alternatives  (Kirkwood,  1997).  By  varying  the  weights 
defined  by  the  decision  maker  and  recalculating  the  additive  function  repeatedly,  the 
impacts  of  changes  in  the  weighting  scheme  on  the  overall  decision  problem  can  be 
determined.  In  other  words,  sensitivity  analysis  can  show  how  the  decision  changes  in 
response  to  changes  in  the  model,  identify  unnecessary  values  or  measures,  and  highlight 
measures  that  it  would  be  wise  to  analyze  closely 

Sensitivity  analysis  was  initially  perfonned  on  the  first-tier  values 
(Capability/Risk,  Cost,  and  Past  Perfonnance).  As  the  weight  assigned  to  a  particular 
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value  was  allowed  to  range  from  0  to  1 ,  the  global  weights  for  the  other  values  on  the 
same  tier  were  maintained  proportionally  constant.  The  sensitivity  graphs  presented  in 
this  section  assign  differing  line  types  based  on  the  alternative’s  rank  when  the  global 
weight  is  1.000;  this  ranking  and  the  associated  line  types  are  displayed  beside  the  right- 
side  vertical  axis. 

Figure  25  shows  the  result  of  varying  the  global  weight  assigned  to  the 
Capability/Risk  value.  Recall  that  at  the  initial  weight  of  0.500,  Alternative  6  was  ranked 
highest.  From  this  graph,  it  can  be  concluded  that  the  decision  is  sensitive  to  these 
changes  in  weight;  however,  it  would  take  a  decrease  in  weight  from  0.500  to 
approximately  0.320  to  change  the  decision  from  Alternative  6  to  Alternative  9. 
Conversely,  it  would  take  an  increase  in  weight  to  approximately  0.720  for  the  decision 
to  change  to  Alternative  1 .  Since  the  weights  for  the  first-tier  values  were  most  likely 
appropriately  distributed  by  the  PDM,  the  weight  assigned  to  Capability/Risk  will 
probably  not  vary  by  the  amounts  necessary  to  change  the  decision.  Therefore,  from  a 
practical  viewpoint,  the  decision  is  not  considered  very  sensitive  to  the  weighting  of 
Capability/Risk  and  the  PDM  does  not  need  to  make  a  detailed  reexamination  of  the 
weight  placed  on  this  value. 

As  evidenced  by  its  steeply  sloping  line,  the  ranking  of  Alternative  9  is  very 
sensitive  to  changes  in  the  weight  assigned  to  the  Capability/Risk  value.  This  would 
indicate  that  the  scores  Alternative  9  received  in  this  area  may  warrant  further 
examination;  it  also  confirms  the  observations  made  from  the  deterministic  analysis. 
Another  observation  is  that  an  increase  in  weight  generally  only  serves  to  further 
differentiate  Alternatives  6  and  1  from  the  rest  of  the  group  in  terms  of  value  earned.  If 
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the  PDM  decided  that  Capability/Risk  deserved  more  weight  or  was  the  only  first-tier 
value  that  should  be  considered,  Alternatives  1  and  6  may  be  the  only  alternatives  to 
consider  from  this  data  set. 
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: 

Alternative  #1 

-  Alternative  #6 
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-  Alternative  #10 

- Alternative  #5 

- Alternative  #4 
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. 

Alternative  #2 

-  Alternative  #8 

-  Alternative  #7 

- Alternative  #9 

- Alternative  #3 

Worst 

0  '  '  ' 

1  1  1  100 

Percent  of  Weight  on 

Capability/Risk  Value 

Figure  25.  Sensitivity  Graph  for  Capability/Risk  Value 


Figure  26  shows  the  result  of  varying  the  global  weight  assigned  to  the  Cost 
value,  which  is  an  example  of  how  a  value  or  measure  might  be  sensitive  to  weight 
changes  in  one  direction  but  insensitive  to  weight  changes  in  the  other  direction.  From 
this  graph,  it  can  be  seen  that  the  decision  is  sensitive  to  a  decrease  in  weight,  although 
the  decision  does  not  switch  from  Alternative  6  to  Alternative  1  until  the  global  weight 
declines  from  0.250  to  approximately  0.130.  However,  the  decision  is  insensitive  to  an 
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increase  in  weight  since  increasing  the  weight  does  not  switch  the  decision  at  any  point. 
Because  the  weights  on  the  first-tier  values  are  not  likely  to  be  altered  drastically,  these 
observations  indicate  that  the  PDM  should  not  devote  a  great  deal  of  time  and  effort  to 
making  minute  changes  to  the  cost  weight. 

Figure  26  also  shows  that  as  the  weight  on  the  Cost  value  approaches  0.000,  the 
alternative  scores  are  relatively  tightly  bunched  together  in  terms  of  value.  Conversely, 
as  the  weight  approaches  1.000,  there  is  noticeably  more  spreading  of  the  value.  This 
shows  that  cost  differentiates  between  the  alternatives  very  well;  therefore,  the  scores  for 
the  measure  under  the  Cost  branch  should  be  looked  at  closely.  By  observing  the  slope 
of  the  lines  for  Alternatives  7  and  10,  it  can  be  seen  that  their  ranks  are  very  sensitive  to 
changes  in  the  weight  on  the  Cost  value.  This  would  indicate  that  the  scores  for  these 
two  alternatives  may  warrant  further  examination. 
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-  Alternative  #2 

-  Alternative  #6 

-  Alternative  #7 

- Alternative  #9 

- Alternative  #5 

-  Alternative  #1 

-  Alternative  #8 

-  Alternative  #3 

- Alternative  #4 
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Worst  ||  I  I  I  I  I  I  I 

0  100 


Percent  of  Weight  on  Cost  Value 


Figure  26.  Sensitivity  Graph  for  Cost  Value 


Figure  27  shows  the  result  of  varying  the  global  weight  assigned  to  the  Past 
Performance  value.  From  this  graph,  it  can  be  seen  that  the  decision  is  insensitive  to 
decreases  in  the  weight  but  sensitive  to  increases  in  the  weight.  An  increase  in  the  global 
weight  from  0.250  to  approximately  0.390  changes  the  decision  from  Alternative  6  to 
Alternative  9;  when  the  weight  is  increased  to  approximately  0.750,  the  decision  changes 
to  Alternative  3.  Despite  the  increases  in  weight  resulting  in  two  changes  to  the 
recommended  decision,  the  PDM  should  not  devote  a  great  deal  of  time  and  effort  to 
making  minute  changes  to  the  cost  weight  since  its  weight  is  not  likely  to  be  altered 
drastically  by  the  PDM. 

Figure  27  also  shows  that  only  Alternatives  3  and  9  increase  in  value  with  an 
increase  in  weight  on  the  Past  Performance  value  and  differentiate  themselves  from  the 
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other  alternatives.  This  may  warrant  additional  investigation  to  ascertain  what  is  causing 
the  differentiation.  The  rankings  of  Alternatives  3  and  5  appear  to  be  the  most  sensitive 
to  changes  in  the  weighting,  which  would  also  indicate  that  additional  investigation  may 
be  needed. 
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Alternative  #3 

-  Alternative  #9 

-  Alternative  #  1 
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Figure  27.  Sensitivity  Graph  for  Past  Performance  Value 


The  sensitivity  graph  for  the  Quality  value,  Figure  28,  shows  another  scenario  that 
might  be  encountered.  In  this  case,  the  decision  is  completely  insensitive  to  any  change 
in  weight  on  the  Quality  value;  Alternative  6  remains  the  best  alternative  no  matter  how 
the  weight  on  the  value  is  varied.  This  can  be  an  indicator  that  a  value  is  not  necessary  to 
evaluate  the  decision  for  the  set  of  alternatives  being  examined.  The  graph  also  shows 
that  the  rank  of  Alternative  5  is  very  sensitive  to  changes  in  the  weight  on  the  Quality 
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value.  If  the  global  weight  of  the  Quality  value  was  increased  to  1.000  then  it  would  be  a 
very  close  second  to  Alternative  6  in  ranking.  This  would  indicate  that  the  scores  for 
alternative  5  may  need  to  be  more  thoroughly  examined.  It  can  also  be  seen  that 
increasing  the  weight  on  Quality  to  1 .000  severely  differentiates  the  top  three  alternatives 
(6,  5,  and  1)  from  the  rest.  This  may  be  another  indication  that  the  scores  obtained  on  the 
measures  under  quality  may  warrant  further  study. 
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Figure  28.  Sensitivity  Graph  for  Quality  Value 


Figure  29  illustrates  that,  in  addition  to  conducting  sensitivity  analysis  on  the 
values  within  the  model,  the  analyst  can  also  conduct  sensitivity  analysis  on  each  specific 
measure.  This  can  be  valuable  in  detennining  which  measures  might  require  more  or  less 
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time  in  terms  of  defining,  gathering  data,  and  scoring.  It  can  also  highlight  measures  that 
may  be  unnecessary  to  the  model. 

Figure  29  shows  that  the  decision  is  completely  insensitive  to  any  change  in 
weight  on  the  “NAC  Information  Gathering  and  Providing  Adequately  Addressed” 
measure;  Alternative  6  remains  the  best  alternative  no  matter  how  the  weight  on  the  value 
is  varied.  The  graph  also  shows  that  the  rank  of  all  ten  alternatives  is  very  insensitive  to 
changes  in  the  weight  on  this  measure.  This  can  be  an  indicator  that  a  measure  is  not 
necessary  to  evaluate  the  decision  for  the  set  of  alternatives  being  examined.  Figure  29 
also  once  again  confirms  the  observations  made  from  the  detenninistic  analysis  about 
Alternative  9. 
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Figure  29.  Sensitivity  Graph  for  NAC  Information  Gathering  and  Providing  Adequately 

Addressed  Measure 
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Chapter  5.  Findings  and  Conclusions 


This  chapter  represents  Step  10  in  the  value  focused  thinking  (VFT)  process  and 
provides  a  short  summary  of  the  results  of  the  research  effort  while  answering  the 
research  questions  that  were  posed  in  Chapter  1 .  It  will  also  present  recommendations 
for  implementation  of  this  model  within  the  utilities  privatization  process,  highlight  the 
strengths  and  weaknesses  of  the  model,  and  identify  avenues  for  further  research  within 
this  field. 


5.1  Process  Overview 

This  VFT  model  was  constructed  for  use  as  a  multiple  objective  decision  analysis 
tool  to  aid  decision  makers  in  selecting  the  best  contractor  during  the  source  selection 
process  of  a  utilities  privatization  effort.  During  this  process  the  ten-step  method  outlined 
by  Shoviak  (2001)  was  used  to  create  the  model  as  well  as  to  answer  the  research 
questions  that  were  presented  in  Chapter  1 .  Recall  from  Chapter  1  that  the  research 
questions  were: 

1 .  What  values  does  the  Air  Force  hold  regarding  utilities  privatization?  What 
measures  can  be  developed  and  used  to  accurately  represent  and  evaluate 
those  values? 

2.  Which  values  affect  the  contract  award  decision  most?  Will  this  hold  true  at 
all  bases  and  throughout  the  process  at  each  base? 

3.  Will  one  set  of  values  hold  true  for  all  utility  systems  (i.e.,  electrical,  water, 
wastewater,  and  natural  gas)?  Can  the  value  hierarchy  be  held  constant  with 
changes  made  only  in  the  measurements,  value  functions,  or  weights  for  each 
system  evaluation? 
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The  first  research  question  was  answered  by  using  the  first  five  steps  of  the  VFT 
process.  The  resulting  hierarchy  consisted  of  three  first-tier  values:  Capability/Risk, 
Cost,  and  Past  Performance.  These  values  were  further  decomposed  into  eight  second- 
tier  values  and  five  third-tier  values.  To  measure  the  level  of  attainment  for  these  values 
26  measures  and  appropriate  value  functions  were  developed.  Local  and  global 
weighting  were  applied  to  both  the  measures  and  values. 

The  second  question  was  answered  by  using  Steps  6  through  9  of  the  VFT 
process.  Once  the  hierarchy  was  developed,  the  generation  and  scoring  of  a  set  of  viable 
alternatives  was  accomplished.  This  was  done  by  creating  a  set  of  500  randomly 
generated  proposals,  using  filters  to  select  ten  of  the  proposals  to  maximize  each  of  the  1 1 
first  and  second-tier  values,  and  scoring  the  proposals.  Detenninistic  and  sensitivity 
analysis  was  then  conducted  on  the  ten  alternatives. 

In  answering  question  two  it  can  be  seen  from  the  detenninistic  and  sensitivity 
analysis  that  the  decision  was  most  affected  by  the  Capability/Risk  value  and  the  “CEA 
NPV  %  of  Should  Cost”  measure.  Examining  the  model  and  associated  global  weights,  it 
is  tempting  to  say  this  is  intuitive;  however,  the  impact  of  the  measures  and  values 
contained  in  the  model  is  very  dependent  on  the  alternatives  evaluated  with  the  model.  A 
different  set  of  proposals  could  (and  probably  will)  provide  completely  different  results 
and  analysis  findings.  Therefore  the  answer  to  the  second  half  of  question  two  is  that 
although  the  impact  of  the  values  and  measures  may  hold  the  same  at  each  base,  it  is  just 
as  possible  that  it  will  not. 

The  third  question  was  answered  by  referring  to  the  research  used  in  creating  the 
hierarchy.  The  Request  for  Proposal  (RFP)  template  distributed  by  the  Air  Force  Civil 
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Engineer  Support  Agency  (AFCESA)  was  created  for  use  with  all  utility  systems.  Since 
the  value  hierarchy  is  heavily  based  upon  the  RFP  template,  it  can  be  used  to  evaluate  all 
utility  systems  with  no  change  to  the  values,  measures,  single  dimension  value  functions 
(SDVFs),  or  weights.  However,  in  order  to  achieve  the  most  helpful  and  insightful 
results,  it  would  be  better  to  reevaluate  the  model  for  use  with  other  systems  and  make 
adjustments  as  necessary.  The  biggest  changes  would  probably  be  to  the  scale  definitions 
for  each  measure  since  they  were  created  specifically  for  electrical  systems. 

5.2  Conclusions 

Two  main  conclusions  can  be  drawn  from  this  research:  VFT  is  a  viable 
technique  for  source  selection  evaluations  and  the  hierarchy  is  general  enough  that  it  can 
be  used  with  a  variety  of  utility  systems  with  very  minor  changes.  VFT  focuses  the 
source  selection  decision  process  on  the  preferences,  opinions,  and  views  of  the  Air  Force 
and  allows  the  quantification  of  the  subjective  evaluation  of  each  proposal.  This  allows 
an  objective  look  at  how  each  alternative  ranks  against  the  others  deterministically;  it  also 
allows  the  decision  maker  to  view  how  sensitive  the  decision  is  to  changes  in  the  model. 
These  insights  provide  the  decision  maker  with  additional  information  that  may  not  have 
been  obvious  simply  by  evaluating  proposals  through  an  alternative  focused  thinking 
perspective.  The  applicability  of  this  model  to  various  utility  systems  implies  that  it  can 
be  integrated  directly  into  the  RFP  process  as  a  secondary  check  on  the  current  evaluation 
process.  After  all,  Orwin  (1999)  stated  that  not  only  is  a  standardized  source  selection 
process  needed,  but  a  backup  source  selection  tool  should  also  be  used  in  order  to  double 
check  the  “accuracy”  of  the  other  process. 
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5.3  Model  Strengths 

This  model  provides  an  economical  means  of  replicating  and  testing  decisions  and 
is  quicker  to  evaluate  than  the  real  life  system  it  represents.  In  addition,  this  model  can 
be  used  to  evaluate  numerous  scenarios;  therefore,  it  is  much  more  feasible  to  implement 
and  use  than  a  trial  and  error  methodology.  This  model  also  requires  the  decision  maker 
to  closely  examine  the  decision  to  be  made  providing  invaluable  insight  and 
understanding  into  the  decision.  It  also  eliminates  bias  and  common  errors  from  the 
decision  by  providing  a  systematic  evaluation  method  and  eliminating  the  tendency  of 
humans  to  “fly  by  the  seat  of  their  pants”  and  use  “rules  of  thumb”  (Post  &  Anderson, 
2003). 

This  VFT  model  also  takes  into  account  the  values  (information,  opinions,  and 
preferences)  of  the  decision  maker  (in  this  case  the  United  States  Air  Force)  (Kirkwood, 
1997).  It  represents  what  the  Air  Force  cares  about  and  what  is  fundamentally  important 
to  this  specific  decision  situation  (Keeney,  1994).  These  values  are  the  driving  force  for 
the  decision  making  process  and  allow  the  Air  Force  to  identify  alternatives  that  more 
completely  and  satisfactorily  match  their  fundamental  values.  Another  particularly 
effective  advantage  of  this  model  is  the  ability  it  gives  the  decision  maker  to  implement 
changes  to  the  model  parameters  to  reflect  minor  differences  such  as  those  found  between 
utility  systems  and  to  reflect  changing  preferences  (such  as  those  of  a  different  decision 
maker).  It  also  allows  easy  inspection  of  the  model’s  sensitivity  to  changes  in  the 
parameters;  this  is  particularly  useful  in  situations  such  as  utilities  privatization  where  the 
impact  of  the  decision  is  so  tremendous  and  maximum  insight  is  desired. 
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5.4  Model  Limitations 


This  model  also  has  several  limitations.  One  major  limitation  of  this  model  is  that 
it  was  developed  using  a  single  proxy  decision  maker.  Although  the  VFT  process  and  the 
use  of  doctrinal  publications  to  facilitate  hierarchy  development  helped  reduce  the 
introduction  of  bias  into  the  model,  having  a  single  proxy  tends  to  create  bias.  If  the 
model  is  used  with  real  world  proposals,  the  scoring  of  the  proposals  also  inherently 
introduces  some  bias  into  the  outcome.  However,  since  a  VFT  model  is  only  meant  to 
provide  insight  and  aid  the  decision  maker  in  choosing  an  alternative,  the  small  amount 
of  bias  present  in  the  model  can  be  handled  provided  the  decision  maker  realizes  it  is 
present. 

Another  possible  limitation  present  in  the  model  is  the  inclusion  of  cost  as  a  value 
within  the  hierarchy.  A  potentially  more  useful  way  to  analyze  proposals  during  a  source 
selection  would  be  to  evaluate  cost  separate  from  the  rest  of  the  hierarchy  and  then  do  a 
cost/benefit  ratio  analysis  after  scoring  is  complete.  This  could  provide  the  decision 
maker  with  a  clearer  picture  of  the  tradeoffs  between  the  potential  benefits  of  the 
privatization  effort  with  the  potential  cost. 

Another  area  that  could  be  improved  within  the  model  is  the  measure(s)  under  the 
second-tier  value  of  Competency.  When  developing  the  model,  the  original  intent  was  to 
have  a  separate  measure  of  “%  of  Positive  Past  Perfonnance  Surveys”  for  each  major 
category  of  evaluation  within  the  past  perfonnance  surveys  used  during  source  selection. 
However,  because  no  source  of  a  past  perfonnance  survey  could  be  found,  the  single 
measure  was  used.  Added  accuracy/benefit  could  be  gained  by  including  separate 
measures  for  each  past  perfonnance  survey  category. 
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5.5  Areas  for  Future  Work 


One  plausible  area  for  future  work  would  be  gaining  additional  validation  or 
adjustments  for  the  model  from  the  Air  Force  Civil  Engineer  Support  Agency 
(AFCESA),  the  Defense  Energy  Support  Center  (DESC),  or  senior  Air  Force  leadership. 
This  would  help  to  ensure  the  model  truly  incorporates  the  values  of  the  Air  Force  with 
regards  to  utilities  privatization  and  provide  additional  confidence  in  the  insight  provided 
by  the  model.  Another  area  for  future  work  would  be  working  with  AFCESA  to 
implement  this  source  selection  evaluation  tool  into  the  current  source  selection  process. 
This  would  be  extremely  beneficial  to  the  Air  Force  by  providing  them  with  a  second 
source  of  insight  into  source  selection  decisions  that  is  objective,  repeatable,  and 
defendable.  A  third  avenue  for  future  work  would  be  analyzing  actual  source  selection 
decisions  using  this  tool.  This  would  not  only  allow  deterministic  analysis  and  sensitivity 
analysis  to  be  performed  on  those  particular  proposals,  but  it  would  also  allow  a 
comparison  of  the  insight  gained  from  the  current  source  selection  process  with  the 
process  represented  by  this  model.  From  this  comparison,  additional  improvements 
could  be  determined  for  each  process. 
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Appendix  A.  Value  Hierarchy  Definitions 


Capability/Risk 

The  Capability/Risk  value  and  the  sub-values  identified  in  that  branch  are  concerned  with 
the  capability  of  the  contractor  as  portrayed  in  the  contractor’s  proposal  and  the 
feasibility  of  the  plans,  work,  and  costs  outlined  in  the  proposal.  This  is  extremely 
important  to  the  Air  Force  since  one  of  the  core  reasons  behind  privatization  is  to 
increase  the  quality  and  efficiency  of  service  from  the  utility  system  by  utilizing  private 
sector  expertise,  while  at  the  same  time  maintaining  or  improving  upon  current  service 
levels.  Capability/Risk  is  crucial  to  the  utilities  privatization  decision  because  a  proper 
assessment  of  how  well  a  contractor  meets  the  criteria  of  this  value  allows  the  source 
selection  authority  to  accurately  award  to  the  firm  that  will  meet  the  goals  of  the 
privatization  initiative.  Capability/Risk  is  further  broken  down  into  5  sub-values: 
Economic  Feasibility;  Quality;  Reliability;  Service,  Service  Interruption,  and 
Contingency  Responsiveness;  and  Smooth  Transition. 

Economic  Feasibility 

Economic  feasibility  is  concerned  with  the  financial  condition  of  the  contractors 
being  evaluated  and  whether  or  not  that  condition  will  affect  the  status  of  the  contract,  the 
performance  of  the  work,  or  the  stability  of  the  service.  This  is  of  great  concern  to  the 
Air  Force  in  evaluating  a  contractor  because  each  contractor  needs  to  have  a  well 
developed  understanding  of  the  economic  risk  associated  with  the  undertaking  and  the 
Air  Force  needs  to  have  some  assurance  that  the  contractors  have  that  understanding. 
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This  is  also  extremely  valuable  to  the  USAF  because  they  must  be  able  to  predict  the 
economic  stability  of  the  contractors  with  some  degree  of  certainty  in  order  to  alleviate 
the  possibility  of  bankruptcy,  low  reinvestment  in  capital  upgrades,  lack  of  new 
technology  use,  and  other  problems  associated  with  poor  financial  condition  of  a 
contractor. 

Quality 

Quality  is  a  reflection  of  the  Air  Force’s  focus  on  retaining  or  improving  actual 
service  levels  with  the  implementation  of  a  utilities  privatization  contract.  Service  levels 
can  be  affected  by  many  things  such  as  governmental  laws  at  all  levels,  local  (base) 
requirements,  management  practices,  worker  safety,  and  system  security.  All  of  these 
items  and  quality  of  service  as  a  whole  is  very  important  in  a  utilities  privatization 
contract,  because  if  the  contractors  provide  the  service  but  fail  to  meet  the  action  items 
listed  above,  then  the  base  can  face  severe  penalties,  increased  costs  and  inefficiency,  loss 
of  lives,  and  potential  loss  of  service.  These  are  conditions  that  the  Air  Force  can  not 
afford,  and,  therefore,  the  USAF  has  to  have  a  firm  evaluation  of  a  contractor's  ability  to 
provide  quality  service  along  with  a  cheaper  product.  The  quality  sub-value  is  further 
broken  down  into  six  sub-sub-values:  Compliance  with  Federal,  State,  Interstate,  and 
Local  Laws  and  Base  Requirements,  Continued  USAF  Readiness,  Effective  Base 
Interaction,  Effective  Management,  Safety,  and  Security. 

Compliance  with  Federal,  State,  Interstate,  and  Local  Laws  and  Base 

Requirements 

This  value  is  concerned  with  a  contractor  meeting  all  laws  that  affect  the 
operation  of  the  privatized  system,  maintaining  proper  licensing,  permitting,  and 


106 


certification  processes,  and  adhering  to  all  base  specific  requirements  such  as  controlled 
access  areas,  the  right  of  way,  training  of  military  personnel,  and  metering.  This  value  is 
understandably  important  because  it  can  have  such  a  tremendous  impact  on  the  base’s 
public  relations,  the  amount  of  fines  levied  against  the  base,  system  operations, 
contractor/govemment  relations,  wartime  readiness,  and  base  expenditures.  Without 
proper  evaluation  of  this  area,  the  system  could  become  a  tremendous  liability  in  the 
future. 

Effective  Base  Interaction 

Due  to  the  importance  of  a  utility  system  to  the  base  infrastructure  system,  base 
employees,  and  base  mission,  interaction  between  the  contractor  and  base  organizations 
such  as  Civil  Engineering  is  critical.  Items  such  as  the  locating  of  utility  lines  and 
submetering  are  crucial  daily  taskings  whose  accuracy  is  relied  upon  by  not  only  base 
units,  but  also  other  contractors  performing  work  on  the  base.  The  Base  Civil  Engineer  is 
also  reliant  on  the  contractor  to  provide  updated  maps  and  drawings  of  new  system 
components  such  as  facilities  and  lines.  Without  the  proper  locating  of  lines  and 
updating  of  maps  and  drawings,  a  base  or  large  areas  of  the  base  can  be  left  without 
electricity,  water,  or  natural  gas  for  days,  workers  can  be  injured  or  killed,  and  the 
mission  of  the  base  can  be  critically  impacted. 

Effective  Management 

Effective  management  is  concerned  with  ensuring  that  the  contractor  employs 
management  practices  that  will  lead  to  the  increased  efficiency  and  service  levels 
expected  from  the  privatization  effort.  Although  by  itself  effective  management  does  not 
determine  the  success  of  a  contractor  in  operating  a  system,  it  is  a  linchpin  of  an 
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operation  and  a  key  indicator  of  a  contractor’s  potential  perfonnance.  This  is  important 
to  the  Air  Force  as  an  indicator  of  a  firm’s  ability  to  take  on  a  utilities  privatization 
partnership  with  a  base. 

Safety 

This  value  focuses  on  the  safety  of  the  workers  that  will  be  operating, 
maintaining,  and  managing  the  utility  system  in  question.  Safety  issues  such  as  proper 
certification  of  the  firm’s  employees  and  the  hiring  of  workers  with  proper  English  skills 
for  the  positions  filled  are  important  simply  for  the  protection  of  human  lives.  These 
actions  also  help  protect  equipment  and  facilities  as  well  as  improving  customer  service 
and  interaction  with  base  personnel  and  organizations. 

Security 

Security  is  an  enonnous  consideration  with  a  utilities  privatization  contract 
because  a  contractor  with  non-military,  non-government  personnel  will  now  be  allowed 
base  access  every  day  in  order  to  operate  a  mission  essential  utility  system.  Although  the 
Air  Force  will  impose  access  restrictions  on  the  contractor  personnel,  the  most  effective 
way  to  prevent  security  incidents  is  proper  screening  of  potential  employees.  Therefore 
potential  contractors  have  to  be  evaluated  on  their  plans  for  helping  to  maintain  the 
security  of  a  base  as  well  as  helping  the  USAF  properly  screen  employees  through 
National  Agency  Checks  and  routine  records  checks. 

Reliability 

Reliability  is  extremely  important  to  the  Air  Force  because  of  the  significant 
impact  that  intennittent  delivery  of  utility  services  would  have  on  an  Air  Force  Base’s 
personnel,  facilities,  activities,  and  overall  mission.  Since  an  Air  Force  Base  is 
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essentially  a  self-contained  small  town  that  is  completely  reliant  on  the  utility  systems 
that  support  it,  reliability  of  those  systems  is  crucial  the  base  as  a  whole.  Increased 
reliability  is  one  of  the  core  reasons  behind  the  push  for  utilities  privatization  and 
therefore  is  an  integral  piece  of  the  source  selection  process.  The  Air  Force  needs  to  have 
an  accurate  picture  of  how  a  contractor  will  affect  the  system  reliability  and  how  they 
plan  to  upgrade  the  system  as  well  as  perform  renewals  and  replacements  of  the  system 
components.  Having  a  grasp  on  these  plans  allows  the  Air  Force  to  develop  a  fairly 
accurate  picture  of  how  reliability  will  be  impacted. 

Service,  Service  Interruption,  and  Contingency  Responsiveness 

This  value  not  only  focuses  on  a  contractors  plans  and  ability  to  maintain,  operate, 
and  repair  the  system  under  normal  circumstances,  but  also  under  austere  conditions  such 
as  natural  disasters,  major  component  failures,  hazardous  substance  spills,  cease  and 
desist  Notices  of  Violations,  and  other  situations  requiring  intense  and  often  specialized 
procedures.  It  also  looks  at  prevention  of  these  occurrences  through  actions  such  as 
minimization  of  hazardous  materials  used  and  hazardous  waste  to  dispose  of.  Accurately 
predicting  how  well  a  contractor  will  perform  in  this  category  is  very  important  to  the  Air 
Force  because  without  effective  operational  plans  for  dealing  with  these  situations  and 
preventing  them,  the  impact  that  these  disasters  have  can  become  debilitating  to  the  base 
and  its  mission  and  the  potential  for  damage  to  equipment  and  facilities  and  loss  of  life  is 
greatly  magnified. 

Smooth  Transition 

Turnover  of  a  utilities  system  and  its  operation  involves  a  great  deal  of 
coordination  and  this  value  reflects  the  desire  of  the  Air  Force  to  know  that  the  contractor 
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will  be  able  to  efficiently  transition  to  ownership  in  a  timely  fashion.  By  evaluating  a 
firm’s  plans  for  an  adequate  time-phased  turnover  of  the  facilities,  equipments,  permits, 
and  operation  and  maintenance  as  well  as  the  responsibility  for  new  construction,  meter 
installation,  meter  reading  and  billing,  and  personnel  hiring  the  Air  Force  hopes  to 
achieve  an  accurate  appraisal  of  how  much  impact  the  transition  will  have  on  base 
operations  and  for  how  long  that  impact  will  occur. 

Cost 

The  cost  value  is  focused  on  the  Net  Present  Value  (NPV)  of  the  stream  of  cash 
flows  that  the  Air  Force  would  be  expected  to  make  to  the  contractor  over  the  lifetime  of 
the  contract.  Once  again,  this  is  one  of  the  top  reasons  for  privatizing  a  system.  Along 
with  the  expectation  of  increased  service  levels,  a  drop  in  cost  is  also  associated  with  the 
utilization  of  a  private  firm’s  greater  expertise  and  efficiency.  This  is  intuitively 
important  to  the  Air  Force  because  like  any  organization  a  base  has  a  limited  amount  of 
funding  available  for  operations  and  therefore  every  dollar  needs  to  be  efficiently 
utilized. 

Past  Performance 

Past  Performance  is  a  top  value  because  it  is  a  good  indicator  of  how  confident 
the  Air  Force  can  be  that  the  contractor  will  perfonn  as  promised  based  on  current  and 
previous  (within  the  past  5  years)  contract  efforts.  This  confidence  level  is  very 
important  due  to  the  length  (pennanency)  of  most  utilities  privatization  contracts  and 
should  reflect  the  apparent  experience  in  owning,  operating,  or  maintaining  utility 
systems  of  similar  size  and  complexity  as  the  system  included  in  the  RFP.  The  Air  Force 
must  have  some  demonstrable  assurance  that  the  contractor  can  do  what  they  say  they 
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can.  Past  performance  is  further  broken  down  into  three  sub-values:  Competency, 
Recency,  and  Relevancy 

Competency 

One  factor  that  the  Air  Force  desires  to  have  demonstrated  in  a  contractor’s  past 
performance  evaluations  is  competence  in  operating,  maintaining,  and  repairing  utility 
systems.  The  USAF  needs  to  have  confidence  in  the  fact  that  the  contractor  will  perform 
competently  and  one  available  gauge  of  this  is  how  well  they  have  perfonned  in  other 
similar  situations. 

Recency 

The  Air  Force  not  only  values  how  well  a  contractor  has  performed  on  past  jobs, 
but  also  how  recently  they  have  performed  contracts  of  similar  scope.  This  is  important 
because  it  shows  recognition  that  the  personnel  base  of  a  firm  is  always  in  transition. 

Like  most  sectors  in  the  private  world,  utilities  are  in  constant  flux  with  their  personnel, 
technology,  and  infrastructure.  A  contractor  that  was  a  top  perfonner  ten  years  ago,  may 
not  be  a  viable  choice  today. 

Relevancy 

Along  with  the  competence  and  recentness  of  a  contractors  past  performance,  the 
Air  Force  values  the  relevancy  of  a  firm’s  past  performance.  This  is  important  because 
even  though  a  contractor  may  be  very  large,  experienced,  and  competent,  they  may  not 
have  the  experience  and  competence  in  operating  a  utility  system.  This  is  a  vital 
distinction  to  make,  because  the  criticality  of  utility  systems  to  an  Air  Force  base  means 
that  the  contractor  has  to  have  the  immediate  know-how  to  run  the  system  or  be  faced 
with  an  extremely  steep  learning  curve  in  order  to  avoid  failure. 
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Appendix  B.  Measure  Definitions 


Ml:  Interest  Coverage 


Label 

Value 

Unqualified 

0.000 

Conditionally  Qualified 

0.600 

Unconditionally  Qualified 

1.000 

Category 

Category  Definition 

Value 

Unqualified 

1 .0  x  or  less  for  Investor  Owned  system.  1 .0  x  or  less  for 
Municipal  or  Cooperative  Owned  system. 

0.0 

Conditionally 

Qualified 

1.1 -2. 5  x  for  Investor  Owned  system.  1.1 -1.5  x  for 
Municipal  or  Cooperative  Owned  system. 

0.6 

Unconditionally 

Qualified 

Greater  than  2.5  x  for  Investor  Owned  system.  Greater 
than  1.5  x  for  Municipal  or  Cooperative  Owned  system. 

1.0 
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M2:  Funds  from  Operation  to  Interest  Ratio 


Label 

Value 

Unqualified 

0.000 

Conditionally  Qualified 

[  0.600] 

U  nc  onditionally  Qualifie  d 

|  1.000] 

Category 

Category  Definition 

Value 

Unqualified 

Less  than  1 .0  x  for  Investor  Owned  system.  1 .0  x  or  less 
for  Municipal  or  Cooperative  Owned  system. 

0.0 

Conditionally 

Qualified 

1 . 0-2 . 75  x  for  Investor  Owned  system.  1 . 1  - 1 . 7  x  for 
Municipal  or  Cooperative  Owned  system. 

0.6 

Unconditionally 

Qualified 

Greater  than  2.75  x  for  Investor  Owned  system.  Greater 
than  1 .7  x  for  Municipal  or  Cooperative  Owned  system. 

1.0 
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M3:  Funds  from  Operation  to  Total  Debt  % 


Label 


Value 


Unqualified 
Conditionally  Qualified 
Unconditionally  Qualified 


I  0.000 
I  0.600 
|  1.000 


Category 

Category  Definition 

Value 

Unqualified 

Below  10%  for  Investor  Owned  system.  Below  7%  for 
Municipal  or  Cooperative  Owned  system. 

0.0 

Conditionally 

Qualified 

10-20%  for  Investor  Owned  system.  7-15%  for 

Municipal  or  Cooperative  Owned  system. 

0.6 

Unconditionally 

Qualified 

Greater  than  20%  for  Investor  Owned  system.  Greater 
than  15%  for  Municipal  or  Cooperative  Owned  system. 

1.0 
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M4:  Total  Debt  to  Total  Capital  Ratio 


Label  Value 

Unqualified 
Conditionally  Qualified 
Unconditionally  Qualified 


0.000 

0.600 

TTooo 


Category 

Category  Definition 

Value 

Unqualified 

Above  70%  for  Investor  Owned  system.  Above  80%  for 
Municipal  or  Cooperative  Owned  system. 

0.0 

Conditionally 

Qualified 

50-70%  for  Investor  Owned  system.  70-80%  for 

Municipal  or  Cooperative  Owned  system. 

0.6 

Unconditionally 

Qualified 

Below  50%  for  Investor  Owned  system.  Below  70%  for 
Municipal  or  Cooperative  Owned  system. 

1.0 
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M5:  Disaster  Recovery  Ratio 


Label 


Value 


Unqualified 

0.000 

Conditionally  Qualified 

0.600 

Unconditionally  Qualified 

1.000 

Category 

Category  Definition 

Value 

Unqualified 

Above  40%  for  Investor  Owned  system.  Above  40% 
for  Municipal  or  Cooperative  Owned  system. 

0.0 

Conditionally 

Qualified 

25-40%  for  Investor  Owned  system.  25-40%  for 
Municipal  or  Cooperative  Owned  system. 

0.6 

Unconditionally 

Qualified 

Less  than  25%  for  Investor  Owned  system.  Less  than 
25%  for  Municipal  or  Cooperative  Owned  system. 

1.0 
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M6:  Compliance  with  Federal,  Interstate,  State,  &  Local  Laws  Adequately 
Addressed 


Label  Value 


None 

Minimal 

Good 

Exceptional 


0.000 

tTsoo 

OOP 

1.000 


Category 

Category  Definition 

Value 

None 

Compliance  with  these  laws  is  not  addressed  in  the  proposal. 

0.0 

Minimal 

Proposal  shall  address  all  applicable  federal,  state,  interstate, 
and  local  laws/regulations  that  must  be  complied  with  in 
order  to  provide  this  service  including  those  requirements 
relating  to  health,  safety,  and  the  environment  (RFP  Section 
C.2.1  and  L.6.2). 

0.6 

Good 

Proposal  expands  to  address  specifically  how  each 
applicable  law  will  affect  each  portion  of  the  operation  and 
how  each  facet  will  be  managed  to  avoid  non-compliance. 

0.8 

Exceptional 

Proposal  includes  all  previous  and  also  addresses  plan  for 
modifying  service  practices  as  applicable  laws  are  dropped, 
added,  or  amended.  Should  include  discussion  of  pending 
laws  and  changes  to  laws. 

1.0 
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M7:  Compliance  with  Base  Requirements  Adequately  Addressed 


Label  Value 


None 

Minimal 

Good 

Exceptional 


0.000 


0.600 

OOP 

1.000 


Category 

Category  Definition 

Value 

None 

Compliance  with  base  requirements  is  not  addressed  in  the 
proposal. 

0.0 

Minimal 

Proposal  acknowledges  and  addresses  the  most  current 
versions  of  any  base-specific  requirements  as  defined  in  the 
utility  specific  attachment  (Section  J)  of  the  RFP. 

0.6 

Good 

Proposal  expands  to  address  specifically  how  each  base 
specific  requirement  will  affect  each  portion  of  the  operation 
and  how  each  facet  will  be  managed  to  comply  with  said 
requirements. 

0.8 

Exceptional 

Proposal  includes  all  previous  and  also  addresses  plan  for 
modifying  service  practices  for  any  requirements  that  are 
not  being  met  and  if  any  new  or  changed  requirements  are 
not  being  met  in  the  future.  Should  include  discussion  of 
pending  requirements  and  changes  to  requirements. 

1.0 
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M8:  Licensing,  Permitting,  &  Certification  Adequately  Addressed 


Label  Value 


None 

Minimal 

Good 

Exceptional 


0.000 

0.600 

OOP 

1.000 


Category 

Category  Definition 

Value 

None 

Acquisition  and  transferal  of  necessary  licenses,  permits, 
and  certificates  is  not  addressed  in  the  proposal. 

0.0 

Minimal 

Proposal  acknowledges  and  addresses  the  requirement  to 
obtain  and  maintain  current  any  and  all  licenses,  pennits,  or 
certifications  necessary  to  own,  maintain,  and  operate  the 
utility  system  (RFP  Sections  C.3.1,  C.10.1,  and  C.13.5). 

0.6 

Good 

Proposal  expands  to  address  specifically  each  pennit, 
license,  and  certificate  and  which  areas  of  the  operation  are 
affected  by  such. 

0.8 

Exceptional 

Proposal  includes  all  previous  and  also  addresses  plan  for 
modifying  service  practices  for  any  permits,  licenses,  or 
certificates  that  cannot  be  obtained  or  have  been  revoked. 
Should  include  discussion  of  any  pending  requirements  and 
changes  to  requirements. 

1.0 
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M9:  Line  Locating  and  Updating  of  Maps  and  Drawings  Adequately  Addressed 


Label  Value 


None 

Minimal 

Good 

Exceptional 


0.000 

0.600 

0.800 

1.000 


Category  Definition 

Value 

None 

Requirement  to  locate  utility  lines,  adhere  to  the  base 
excavation  permit  process,  and  maintain  and  provide  record 
drawings  not  addressed  in  the  proposal. 

0.0 

Minimal 

Proposal  acknowledges  and  addresses  the  requirements  to 
locate  utility  lines,  adhere  to  the  base  excavation  permit 
process,  and  maintain  and  provide  record  drawings  (RFP 
Sections  C.5.1,  C.9.5,  and  C.9.6). 

0.6 

Good 

Proposal  expands  to  address  specific  plan  of  action  and 
responsibilities  for  locating  utility  lines,  conforming  to  base 
excavation  permit  process,  and  maintaining  and  providing 
record  drawings. 

0.8 

Exceptional 

Proposal  includes  all  previous  and  also  addresses  proposed 
time  requirements/standards  that  contractor  will  meet  for 
each  activity. 

1.0 
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M10:  Best  Engineering  and  Management  Practices  Adequately  Addressed 


Label  Value 


None 

Minimal 

Good 

Exceptional 


0.000 

0.600 

OOP 

1.000 


Category 

Category  Definition 

Value 

None 

Use  of  Best  Engineering  and  Management  Practices  not 
addressed  in  the  proposal. 

0.0 

Minimal 

Proposal  acknowledges  and  addresses  the  requirement  to 
utilize  Best  Engineering  and  Management  Practices 
consistent  with  Publications  listed  in  RFP  Section  L.6.2  to 
include  a  quality  philosophy  based  on  accepted  management 
practices  and  a  proven  system  of  inspections  and  other 
quality  assessment  procedures  and  techniques,  procedures, 
and  perfonnance  metrics  and  standards.  (RFP  Sections 

L.6.2  and  M.4.2.2). 

0.6 

Good 

Proposal  expands  to  address  specifically  how  their  quality 
philosophy  will  be  applied  to  daily  operations  of  the  system. 

0.8 

Exceptional 

Proposal  includes  all  previous  and  also  addresses  plan  for 
modifying  service  practices  to  initiate  and  facilitate 
continuous  improvement  processes. 

1.0 

121 


Mil:  Level  of  Awards  and  Certificates 


Label  Value 


None 

Local 

State 

National 


0.000 

Poo 

Poo 

TTooo 


Category 

Category  Definition 

Value 

None 

No  Quality  Awards  or  Quality  Certificates  received  at  any 
level  (RFP  Sections  L.6.2  and  M.4.2.2). 

0.0 

Local 

Quality  Awards  or  Certificates  received  at  the  local  level 
(highest  level). 

0.6 

State 

Quality  Awards  or  Certificates  received  at  the  state  level 
(highest  level). 

0.8 

National 

Quality  Awards  or  Certificates  received  at  the  national  level 
(highest  level). 

1.0 
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M12:  Certification  of  Employees  Adequately  Addressed 


Label  Value 

Yes 
No 


1.000 

Moo 


Category  Definition 

Value 

No 

Certification  of  employees  hired  by  the  contractor  is  not 
addressed  in  proposal. 

0.0 

Yes 

Proposal  addresses  certification  of  employees  including 
discussions  on  how  they  will  ensure  proper  skill  levels  of 
workers  and  what  training  and  certifications  are  required  for 
each  job. 

1.0 
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M13:  English  Skills  of  Employees  Adequately  Addressed 


Label  Value 

Yes 
No 


1.000 

Ml 


Category  Definition 

Value 

No 

English  skills  of  employees  hired  by  the  contractor  are  not 
addressed  in  proposal. 

0.0 

Yes 

Proposal  addresses  English  skills  of  employees  including 
discussions  on  how  they  will  ensure  proper  skill  levels  of 
workers  and  what  skill  levels  are  required  for  each  job. 

1.0 
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M14:  National  Agency  Check  Information  Gathering  and  Providing  Adequately 


Addressed 


Label  Value 


Yes 

No 


1.000 

ooo 


Category 

Category  Definition 

Value 

No 

The  gathering  and  providing  of  NAC  information  on 
employees  hired  by  the  contractor  is  not  addressed  in 
proposal. 

0.0 

Yes 

Proposal  addresses  the  procedures  that  will  be  used  for 
gathering  and  providing  NAC  information  on  employees 
hired  by  the  contractor. 

1.0 
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M15:  Capital  Upgrades  and  Renewals  and  Replacements  Plan  Adequacy 


Label  Value 


None 

Minimal 

Good 

Exceptional 


0.000 

0.600 

OOP 

1.000 


Category 

Category  Definition 

Value 

None 

Capital  Upgrades  and  Renewals  and  Replacements  Plan  not 
included  in  the  proposal. 

0.0 

Minimal 

Proposal  describes  in  detail  the  purpose,  scope,  and  benefit 
of  the  initial  renewals  and  replacements  as  well  as  a  detailed 
description  of  the  contractor’s  philosophy  towards  long-tenn 
capital  upgrades  and  renewals  to  include  as  a  minimum  the 
items  listed  in  RFP  Section  L.6.3  (Also  ref.  RFP  Sections 
C.11.2  and  C.  13). 

0.6 

Good 

Proposal  expands  to  address  specific  plans  for  upgrades  that 
will  increase  the  overall  efficiency  of  the  system. 

0.8 

Exceptional 

Proposal  includes  all  previous  and  exceeds  requirements  in  a 
way  beneficial  to  the  government. 

1.0 
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M16:  Hazardous  Materials  and  Hazardous  Waste  Minimization  Adequately 
Addressed 


Label  Value 


None 

Minimal 

Good 

Exceptional 


0.000 

0.600 

0.800 

1.000 


Category 

Category  Definition 

Value 

None 

Hazardous  materials  and  waste  minimization  not  addressed 
in  the  proposal. 

0.0 

Minimal 

Proposal  acknowledges  and  addresses  the  requirement  to 
handle  hazardous  materials  in  accordance  with  applicable 
laws  and  regulations  and  RFP  Sections  C.10.3  and  H.8. 

0.6 

Good 

Proposal  expands  to  identify  non-hazardous  or  less 
hazardous  materials  than  those  currently  in  use  that 
contractor  proposes  to  use. 

0.8 

Exceptional 

Proposal  includes  all  previous  and  also  gives  detailed  plans 
for  consumable  hazardous  waste  recycling  plan. 

1.0 
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M17:  Service  Providance  Adequately  Addressed 


Label  Value 


None 

Minimal 

Good 

Exceptional 


0.000 


0.600 

0^00 

TTooo 


Category 

Category  Definition 

Value 

None 

Service  requests  and  responses,  connections  and 
disconnections,  and  service  interruptions  are  not  addressed 
in  the  proposal. 

0.0 

Minimal 

Proposal  adequately  addresses  routine,  urgent,  and 
emergency  service  requests,  connections  and 
disconnections,  and  scheduled  service  interruptions  in 
accordance  with  RFP  Sections  C.8  and  C.9. 

0.6 

Good 

Proposal  expands  to  include  listing  of  workarounds  for 
specific  scheduled  service  interruptions. 

0.8 

Exceptional 

Proposal  includes  all  previous  and  exceeds  service 
requirements  in  a  way  beneficial  to  the  government. 

1.0 
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M18:  Spill  Contingency  Plan  Adequacy 


Label  Value 


None 

Minimal 

Good 

Exceptional 


0.000 

0.600 

OOP 

TTooo 


Category 

Category  Definition 

Value 

None 

Spill  contingency  plan  not  included  in  the  proposal. 

0.0 

Minimal 

Proposal  includes  adoption  of  the  Installation  Spill 
Contingency  Plan  or  use  of  a  USAF  accepted,  contractor 
developed  Spill  Contingency  Plan  developed  in  accordance 
with  the  National  Response  Team’s  Integrated  Contingency 
Plan  Guidance. 

0.6 

Good 

Proposal  expands  to  include  listing  of  mission  essential 
workarounds  for  spill  contingency  responses. 

0.8 

Exceptional 

Proposal  includes  all  previous  and  procedures  for  regularly 
scheduled  spill  contingency  exercises. 

1.0 
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M19:  Days  to  Transition 


Selected  Point- 


Level: 


0 


Value:  0. 


M20:  Transition  Plan  Adequacy 


Label  Value 


None 

Minimal 

Good 

Exceptional 


0.000 

ITgoo 

OOP 

1.000 


|  EjfflSBH 

Category  Definition 

Value 

None 

Transition  plan  not  included  in  the  proposal. 

o.o  j 

Minimal 

Proposal  included  transition  plan  addressing  hiring  of  work 
force,  acquisition  of  equipment  and  materials,  operations 
testing,  environmental  surveys,  obtaining  of  permits, 
construction,  maintenance,  improvements,  new  connections 
and  meters,  and  elimination  of  safety  hazards  in  accordance 
with  RFP  Sections  C.13,  L.6.4,  and  M.4.2.4. 

0.6 

Good 

Proposal  expands  to  include  approved  and  coordinated 
weekly  meeting  schedule  with  the  Contracting  Officer’s 
representative  during  transition  period. 

0.8 

Exceptional 

Proposal  includes  all  previous  and  detailed  explanation  of 
how  contractor  proposes  to  specifically  minimize  service 
impacts  for  all  base  organizations. 

1.0 
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M21:  Certified  Economic  Analysis  Net  Present  Value  as  a  %  of  Government 


Should  Cost  Estimate 


M22:  %  of  Positive  Past  Performance  Surveys 


Selected  Point  - 


Level: 


Value:  0.5 


M23:  #  of  Similar  Scope  Systems  in  Last  1-3  Years 
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M24:  #  of  Similar  Scope  Systems  in  Last  4-5  Years 


Selected  Point  — 


Level:  2 


Value: 


M25:  %  of  Similar  Scope  Systems  in  Last  1-3  Years 


Selected  Point- 


Level:  15 


Value:  0. 


M26:  %  of  Similar  Scope  Systems  in  Last  4-5  Years 


Selected  Point -- 


Level:  20 


Value:  0.5 


Appendix  C.  Alternative  Scores 


Alt.  # 

Prop.  # 

Ml 

M2 

M3 

A1 

P89 

Unconditionally  Qualified 

Unconditionally  Qualified 

Conditionally  Qualified 

A2 

P389 

Conditionally  Qualified 

Unqualified 

Conditionally  Qualified 

A3 

P488 

Unconditionally  Qualified 

Conditionally  Qualified 

Unqualified 

A4 

P260 

Unconditionally  Qualified 

Unconditionally  Qualified 

Unconditionally  Qualified 

A5 

P354 

Conditionally  Qualified 

Conditionally  Qualified 

Conditionally  Qualified 

A6 

P285 

Unconditionally  Qualified 

Unconditionally  Qualified 

Unconditionally  Qualified 

A7 

P83 

Unconditionally  Qualified 

Unconditionally  Qualified 

Unconditionally  Qualified 

A8 

P297 

Unconditionally  Qualified 

Conditionally  Qualified 

Unqualified 

A9 

P216 

Unconditionally  Qualified 

Unqualified 

Unqualified 

A10 

P333 

Conditionally  Qualified 

Conditionally  Qualified 

Unqualified 

Alt.# 

Prop.  # 

M4 

M5 

M6 

M7 

A1 

P89 

Unconditionally  Qualified 

Conditionally  Qualified 

Exceptional 

Good 

A2 

P389 

Unqualified 

Unconditionally  Qualified 

Minimal 

Exceptional 

A3 

P488 

Conditionally  Qualified 

Conditionally  Qualified 

Minimal 

Good 

A4 

P260 

Unconditionally  Qualified 

Unconditionally  Qualified 

Good 

Good 

A5 

P354 

Conditionally  Qualified 

Unconditionally  Qualified 

Exceptional 

Good 

A6 

P285 

Unqualified 

Conditionally  Qualified 

Minimal 

Minimal 

A7 

P83 

Unconditionally  Qualified 

Unqualified 

Exceptional 

Minimal 

A8 

P297 

Conditionally  Qualified 

Conditionally  Qualified 

Minimal 

None 

A9 

P216 

Unqualified 

Unqualified 

Minimal 

Good 

A10 

P333 

Unconditionally  Qualified 

Unconditionally  Qualified 

Good 

None 

Alt.  # 

Prop.  # 

M8 

M9 

M10 

Mil 

M12 

M13 

M14 

M15 

A1 

P89 

Exceptional 

None 

Minimal 

National 

Yes 

Yes 

Yes 

Exceptional 

A2 

P389 

Minimal 

None 

Exceptional 

State 

No 

Yes 

No 

Exceptional 

A3 

P488 

None 

Minimal 

Good 

State 

Yes 

Yes 

No 

Minimal 

A4 

P260 

Good 

Good 

Minimal 

None 

Yes 

Yes 

No 

None 

A5 

P354 

Exceptional 

Exceptional 

Exceptional 

State 

Yes 

No 

Yes 

Good 

A6 

P285 

Exceptional 

Good 

Exceptional 

State 

Yes 

Yes 

Yes 

Exceptional 

A7 

P83 

Minimal 

Exceptional 

Good 

Local 

No 

Yes 

No 

None 

A8 

P297 

None 

None 

Good 

National 

Yes 

No 

No 

Minimal 

A9 

P216 

Exceptional 

None 

None 

None 

Yes 

Yes 

No 

Minimal 

A10 

P333 

Minimal 

Good 

Exceptional 

State 

No 

No 

No 

Good 
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Alt.# 

Prop.  # 

M16 

M17 

M18 

M19 

M20 

M21 

M22 

M23 

M24 

A1 

P89 

Exceptional 

Exceptional 

Exceptional 

102 

Exceptional 

92 

68 

2 

4 

A2 

P389 

Minimal 

Exceptional 

Minimal 

244 

None 

75 

68 

5 

5 

A3 

P488 

None 

Minimal 

Minimal 

270 

None 

97 

100 

4 

4 

A4 

P260 

Exceptional 

Exceptional 

Good 

158 

Exceptional 

97 

66 

4 

0 

A5 

P354 

Minimal 

Good 

None 

329 

Good 

86 

85 

0 

1 

A6 

P285 

Exceptional 

Minimal 

Exceptional 

107 

Exceptional 

75 

94 

0 

4 

A7 

P83 

Exceptional 

Exceptional 

Exceptional 

292 

Good 

75 

80 

5 

2 

A8 

P297 

Minimal 

Minimal 

Good 

90 

Exceptional 

93 

80 

4 

4 

A9 

P216 

None 

Exceptional 

Exceptional 

232 

Minimal 

76 

100 

1 

5 

A10 

P333 

Exceptional 

Exceptional 

Exceptional 

96 

Good 

99 

75 

3 

1 

Alt.# 

Prop.  # 

M25 

M26 

A1 

P89 

31 

46 

A2 

P389 

45 

9 

A3 

P488 

46 

28 

A4 

P260 

33 

33 

A5 

P354 

1 

31 

A6 

P285 

20 

23 

A7 

P83 

31 

7 

A8 

P297 

0 

47 

A9 

P216 

27 

44 

A10 

P333 

50 

46 
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